56053534c5
Adds a new SAX-based sitemap parser, inspired by: https://www.npmjs.com/package/sitemap-stream-parser Supports: - recursively parsing sitemap indexes, using p-queue to process N at a time (currently 5) - `fromDate` and `toDate` filter dates, to only include URLs between the given dates, filtering nested sitemap lists included - async parsing, continue parsing in the background after 100 URLs - timeout for initial fetch / first 100 URLs set to 30 seconds to avoid slowing down the crawl - save/load state integration: mark if sitemaps have already been parsed in redis, serialize to save state, to avoid reparsing again. (Will reparse if parsing did not fully finish) - Aware of `pageLimit`, don't add URLs pass the page limit, interrupt further parsing when at limit. - robots.txt `sitemap:` parsing, check URL extension and mime type - automatic detection of sitemaps for a seed URL if no sitemap url provided - first check robots.txt, then /sitemap.xml - tests: test for full sitemap autodetect, sitemap with limit, and sitemap from specific URL. Fixes #496 --------- Co-authored-by: Tessa Walsh <tessa@bitarchivist.net> |
||
|---|---|---|
| .github/workflows | ||
| .husky | ||
| config | ||
| docs | ||
| html | ||
| src | ||
| tests | ||
| .dockerignore | ||
| .eslintignore | ||
| .eslintrc.cjs | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .prettierignore | ||
| CHANGES.md | ||
| docker-compose.yml | ||
| docker-entrypoint.sh | ||
| Dockerfile | ||
| LICENSE | ||
| NOTICE | ||
| package.json | ||
| README.md | ||
| requirements.txt | ||
| test-setup.js | ||
| tsconfig.json | ||
| yarn.lock | ||
Browsertrix Crawler 1.x
Browsertrix Crawler is a standalone browser-based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker container. Browsertrix Crawler uses Puppeteer to control one or more Brave Browser browser windows in parallel. Data is captured through the Chrome Devtools Protocol (CDP) in the browser.
For information on how to use and develop Browsertrix Crawler, see the hosted Browsertrix Crawler documentation.
For information on how to build the docs locally, see the docs page.
Support
Initial support for 0.x version of Browsertrix Crawler, was provided by Kiwix. The initial functionality for Browsertrix Crawler was developed to support the zimit project in a collaboration between Webrecorder and Kiwix, and this project has been split off from Zimit into a core component of Webrecorder.
Additional support for Browsertrix Crawler, including for the development of the 0.4.x version has been provided by Portico.