browsertrix-crawler/src
Ilya Kreymer 2ef8e00268
fix connection leaks in aborted fetch() requests (#924)
- in doCancel(), use abort controller and call abort(), instead of
body.cancel()
- ensure doCancel() is called when a WARC record is not written, eg. is
a dupe, as stream is likely not consumed
- also call IO.close() when uses browser network reader
- fixes #923
- also adds missing dupe check to async resources queued from behaviors
(were being deduped on write, but were still fetched unnecessarily)
2025-11-27 20:37:24 -08:00
..
util fix connection leaks in aborted fetch() requests (#924) 2025-11-27 20:37:24 -08:00
crawler.ts Add downloads dir to cache external dependency within the crawl (#921) 2025-11-26 19:30:27 -08:00
create-login-profile.ts Add downloads dir to cache external dependency within the crawl (#921) 2025-11-26 19:30:27 -08:00
main.ts Add more exit codes to detect interruption reason (#764) 2025-02-10 14:00:55 -08:00
replaycrawler.ts Support downloading seed file from URL (#852) 2025-07-03 10:49:37 -04:00