Tessa Walsh 1d15a155f2 Add option to respect robots.txt disallows (#888) ... Fixes #631 - Adds --robots flag which will enable checking robots.txt for each host for each page, before the page is queued for further crawler. - Supports --robotsAgent flag which configures agent to check in robots.txt, in addition to '*'. Defaults to 'Browsertrix/1.x' - Robots.txt bodies are parsed and checked for page allow/disallow status using the https://github.com/samclarke/robots-parser library, which is the most active and well-maintained implementation I could find with TypeScript types. - Fetched robots.txt bodies are cached by their URL in Redis using an LRU, retaining last 100 robots entries, each upto 100K - Non-200 responses are treated as empty robots, and empty robots are treated as 'allow all' - Multiple request to same robots.txt are batched to perform only one fetch, waiting up to 10 seconds per fetch. --------- Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>		2025-11-26 19:00:06 -08:00
..
docs	Add option to respect robots.txt disallows (#888)	2025-11-26 19:00:06 -08:00
gen-cli.sh	Gracefully handle non-absolute path for create-login-profile --filename (#521)	2024-03-29 13:46:54 -07:00
mkdocs.yml	Update Behaviors Docs (#820)	2025-04-10 03:58:07 -04:00