e5bab8e7c8
- rework 'should stream' logic: * ensure 206 responses (or any response) greater than 25M are streamed * response between 5M and 25M are read into memory if text/css/js as they may be rewritten * responses <5M are read into memory * responses with unknown size are streamed if a 2xx, otherwise read into memory, assuming error code responses may lack status codes but otherwise are small - likely fix for issues in #706 - if too many range requests for same URL are being made, try skipping/failing right away to reduce load - assume main browser context is used not just for service workers, always enable - check false positive 'net-aborted' error that may actually be ok for media, as well as documents - improve logging - interrupt any pending requests (that may be loading via browser context) after page timeout, log dropped requests --------- Co-authored-by: Tessa Walsh <tessa@bitarchivist.net> |
||
|---|---|---|
| .github/workflows | ||
| .husky | ||
| config/policies | ||
| docs | ||
| html | ||
| src | ||
| tests | ||
| .dockerignore | ||
| .eslintignore | ||
| .eslintrc.cjs | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .prettierignore | ||
| CHANGES.md | ||
| docker-compose.yml | ||
| docker-entrypoint.sh | ||
| Dockerfile | ||
| LICENSE | ||
| NOTICE | ||
| package.json | ||
| README.md | ||
| requirements.txt | ||
| test-setup.js | ||
| tsconfig.eslint.json | ||
| tsconfig.json | ||
| yarn.lock | ||
Browsertrix Crawler 1.x
Browsertrix Crawler is a standalone browser-based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker container. Browsertrix Crawler uses Puppeteer to control one or more Brave Browser browser windows in parallel. Data is captured through the Chrome Devtools Protocol (CDP) in the browser.
For information on how to use and develop Browsertrix Crawler, see the hosted Browsertrix Crawler documentation.
For information on how to build the docs locally, see the docs page.
Support
Initial support for 0.x version of Browsertrix Crawler, was provided by Kiwix. The initial functionality for Browsertrix Crawler was developed to support the zimit project in a collaboration between Webrecorder and Kiwix, and this project has been split off from Zimit into a core component of Webrecorder.
Additional support for Browsertrix Crawler, including for the development of the 0.4.x version has been provided by Portico.