Stowage/browsertrix-crawler
2
0
Fork 0
mirror of https://github.com/webrecorder/browsertrix-crawler.git synced 2025-10-19 06:23:16 +00:00
Code Issues Projects Releases Packages Wiki Activity
Run a high-fidelity browser-based web archiving crawler in a single Docker container https://crawler.docs.browsertrix.com
483 commits 54 branches 144 tags 82 MiB
Find a file
Exact
Ilya Kreymer cf3a4dcb44 instead of puppeteer.exposeFunction(), use cdp function bindings directly to avoid issues custom toJSON overrides: 
- add Runtime.addBinding for each exposed function, handle in one place with Runtime.bindingCalled
- convert binding names to BxFunctionBindings enum
- update to browsertrix-behaviors 0.7.1 to avoid waiting for return value
- fixes #770
2025-02-20 20:49:48 -08:00
.github/workflows separate fetch api for autofetch bbehavior + additional improvements on partial responses: (#736) 2024-12-31 13:52:12 -08:00
.husky Add MKDocs documentation site for Browsertrix Crawler 1.0.0 (#494) 2024-03-16 14:59:32 -07:00
config/policies cleanup: remove old config files from pywb (#682) 2024-09-05 20:23:34 -07:00
docs Add documentation on exit codes (#765) 2025-02-11 12:16:29 -05:00
html Adblock support (#534) 2024-04-12 09:47:32 -07:00
src instead of puppeteer.exposeFunction(), use cdp function bindings directly to avoid issues custom toJSON overrides: 2025-02-20 20:49:48 -08:00
tests Add more exit codes to detect interruption reason (#764) 2025-02-10 14:00:55 -08:00
.dockerignore Add ad blocking via request interception (#173) 2022-11-15 18:30:27 -08:00
.eslintignore follow-up to #428: update ignore files (#431) 2023-11-09 17:13:53 -08:00
.eslintrc.cjs eslint: add strict await checking: (#684) 2024-09-06 16:24:18 -07:00
.gitignore Gracefully handle non-absolute path for create-login-profile --filename (#521) 2024-03-29 13:46:54 -07:00
.pre-commit-config.yaml Add Prettier to the repo, and format all the files! (#428) 2023-11-09 16:11:11 -08:00
.prettierignore follow-up to #428: update ignore files (#431) 2023-11-09 17:13:53 -08:00
behaviors.js Autoclick Support (#729) 2025-01-16 09:38:11 -08:00
CHANGES.md Add Prettier to the repo, and format all the files! (#428) 2023-11-09 16:11:11 -08:00
docker-compose.yml Add Prettier to the repo, and format all the files! (#428) 2023-11-09 16:11:11 -08:00
docker-entrypoint.sh clear out core dumps to avoid using up volume space: (#740) 2025-01-16 15:50:59 -08:00
Dockerfile browser: update brave to 1.77.52 to get Chromium 134 (#773) 2025-02-20 09:14:32 -08:00
LICENSE initial commit after split from zimit 2020-10-31 13:16:37 -07:00
NOTICE initial commit after split from zimit 2020-10-31 13:16:37 -07:00
package.json instead of puppeteer.exposeFunction(), use cdp function bindings directly to avoid issues custom toJSON overrides: 2025-02-20 20:49:48 -08:00
README.md Add MKDocs documentation site for Browsertrix Crawler 1.0.0 (#494) 2024-03-16 14:59:32 -07:00
requirements.txt Separate writing pages to pages.jsonl + extraPages.jsonl to use with new py-wacz (#535) 2024-04-11 13:55:52 -07:00
test-setup.js Fix disk utilization computation errors (#338) 2023-07-05 21:58:28 -07:00
tsconfig.eslint.json eslint: add strict await checking: (#684) 2024-09-06 16:24:18 -07:00
tsconfig.json dep: update to wabac.js 2.20 (#704) 2024-10-16 21:02:04 -07:00
yarn.lock instead of puppeteer.exposeFunction(), use cdp function bindings directly to avoid issues custom toJSON overrides: 2025-02-20 20:49:48 -08:00

README.md

Browsertrix Crawler 1.x

Browsertrix Crawler is a standalone browser-based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker container. Browsertrix Crawler uses Puppeteer to control one or more Brave Browser browser windows in parallel. Data is captured through the Chrome Devtools Protocol (CDP) in the browser.

For information on how to use and develop Browsertrix Crawler, see the hosted Browsertrix Crawler documentation.

For information on how to build the docs locally, see the docs page.

Support

Initial support for 0.x version of Browsertrix Crawler, was provided by Kiwix. The initial functionality for Browsertrix Crawler was developed to support the zimit project in a collaboration between Webrecorder and Kiwix, and this project has been split off from Zimit into a core component of Webrecorder.

Additional support for Browsertrix Crawler, including for the development of the 0.4.x version has been provided by Portico.

License

AGPLv3 or later, see LICENSE for more details.