705bc0cd9f
- separate out reading stream response while browser is waiting (not really async) from actual async loading, this is not handled via fetchResponseBody() - unify async fetch into first trying browser networking for regular GET, fallback to regular fetch() - load headers and body separately in async fetch, allowing for cancelling request after headers - refactor direct fetch of non-html pages: load headers and handle loading body, adding page async, allowing worker to continue loading browser-based pages (should allow more parallelization in the future) - unify WARC writing in preparation for dedup: unified serializeWARC() called for all paths, WARC digest computed, additional checks for payload added for streaming loading |
||
|---|---|---|
| .github/workflows | ||
| .husky | ||
| config/policies | ||
| docs | ||
| html | ||
| src | ||
| tests | ||
| .dockerignore | ||
| .eslintignore | ||
| .eslintrc.cjs | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .prettierignore | ||
| .prettierrc | ||
| CHANGES.md | ||
| docker-compose.yml | ||
| docker-entrypoint.sh | ||
| Dockerfile | ||
| LICENSE | ||
| NOTICE | ||
| package.json | ||
| README.md | ||
| requirements.txt | ||
| test-setup.js | ||
| tsconfig.eslint.json | ||
| tsconfig.json | ||
| yarn.lock | ||
Browsertrix Crawler 1.x
Browsertrix Crawler is a standalone browser-based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker container. Browsertrix Crawler uses Puppeteer to control one or more Brave Browser browser windows in parallel. Data is captured through the Chrome Devtools Protocol (CDP) in the browser.
For information on how to use and develop Browsertrix Crawler, see the hosted Browsertrix Crawler documentation.
For information on how to build the docs locally, see the docs page.
Support
Initial support for 0.x version of Browsertrix Crawler, was provided by Kiwix. The initial functionality for Browsertrix Crawler was developed to support the zimit project in a collaboration between Webrecorder and Kiwix, and this project has been split off from Zimit into a core component of Webrecorder.
Additional support for Browsertrix Crawler, including for the development of the 0.4.x version has been provided by Portico.