e2b4cc1844
fixes #587 The proxy env vars PROXY_HOST and PROXY_PORT were being ignored, as they were hardcoded to obsolete values in the Dockerfile. Proxy settings can now be set, in order of precedence via: - --proxyServer cli flag - PROXY_SERVER env var - PROXY_HOST and PROXY_PORT env vars, which set an HTTP proxy server only (for backwards compatibility with 0.12.x) The --proxyServer / PROXY_SERVER settings are passed to the browser via the --proxy-server flag. AsyncFetcher / direct fetch also supports HTTP and SOCKS5 proxying. Supported proxies are: HTTP no auth, SOCKS5 no auth, SOCKS5 with auth (supported in Brave, but not Chrome!) --------- Co-authored-by: Vinzenz Sinapius <Vinzenz.Sinapius@gmail.com> Co-authored-by: Tessa Walsh <tessa@bitarchivist.net> |
||
|---|---|---|
| .github/workflows | ||
| .husky | ||
| config | ||
| docs | ||
| html | ||
| src | ||
| tests | ||
| .dockerignore | ||
| .eslintignore | ||
| .eslintrc.cjs | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .prettierignore | ||
| CHANGES.md | ||
| docker-compose.yml | ||
| docker-entrypoint.sh | ||
| Dockerfile | ||
| LICENSE | ||
| NOTICE | ||
| package.json | ||
| README.md | ||
| requirements.txt | ||
| test-setup.js | ||
| tsconfig.json | ||
| yarn.lock | ||
Browsertrix Crawler 1.x
Browsertrix Crawler is a standalone browser-based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker container. Browsertrix Crawler uses Puppeteer to control one or more Brave Browser browser windows in parallel. Data is captured through the Chrome Devtools Protocol (CDP) in the browser.
For information on how to use and develop Browsertrix Crawler, see the hosted Browsertrix Crawler documentation.
For information on how to build the docs locally, see the docs page.
Support
Initial support for 0.x version of Browsertrix Crawler, was provided by Kiwix. The initial functionality for Browsertrix Crawler was developed to support the zimit project in a collaboration between Webrecorder and Kiwix, and this project has been split off from Zimit into a core component of Webrecorder.
Additional support for Browsertrix Crawler, including for the development of the 0.4.x version has been provided by Portico.