Ilya Kreymer a2742df328 seed urls list: check for quoted URLs and remove quotes (#883) ... - check for urls that are wrapped in quotes, eg. 'https://example.com/' or "https://example.com/" and trim and remove the quotes before adding seed - tests: add quoted URL to tests, fix old.webrecorder.net test - deps: update wabac.js, RWP to latest - logging: reduce error logging for seed lists, only log once that there are duplicates or page limit is reached - fix for #882		2025-09-12 13:34:41 -07:00
..
proxies	Support host-specific proxies with proxy config YAML (#837)	2025-08-20 16:07:29 -07:00
crawl-1.yaml	Add Prettier to the repo, and format all the files! (#428)	2023-11-09 16:11:11 -08:00
crawl-2.yaml	Add fields to warcinfo in combinedwarc (#60)	2021-07-07 15:56:52 -07:00
driver-1.mjs	Support custom css selectors for extracting links (#689)	2024-11-08 11:04:41 -05:00
pages.jsonl	tests text extraction (#30)	2021-03-01 16:00:23 -08:00
urlSeedFile.txt	seed urls list: check for quoted URLs and remove quotes (#883)	2025-09-12 13:34:41 -07:00