Ilya Kreymer 00835fc4f2 Retry same queue (#757) ... - follow up to #743 - page retries are simply added back to the same queue with `retry` param incremented and a higher scope, after extraHops, to ensure retries are added at the end. - score calculation is: `score = depth + (extraHops * MAX_DEPTH) + (retry * MAX_DEPTH * 2)`, this ensures that retries have lower priority than extraHops, and additional retries even lower priority (higher score). - warning is logged when a retry happens, error only when all retries are exhausted. - back to one failure list, urls added there only when all retries are exhausted. - rename --numRetries -> --maxRetries / --retries for clarity - state load: allow retrying previously failed URLs if --maxRetries is higher then on previous run. - ensure working with --failOnFailedStatus, if provided, invalid status codes (>= 400) are retried along with page load failures - fixes #132 --------- Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>		2025-02-06 18:48:40 -08:00
..
docs	Retry same queue (#757)	2025-02-06 18:48:40 -08:00
gen-cli.sh	Gracefully handle non-absolute path for create-login-profile --filename (#521)	2024-03-29 13:46:54 -07:00
mkdocs.yml	Add documentation for crawl collections (#695)	2024-10-05 11:51:32 -07:00