Commit graph

1 commit

Author SHA1 Message Date
Ilya Kreymer
f6edec0b95
Fix for --rolloverSize for individual WARCs in 1.x (#542)
Fixes #533 

Fixes rollover in WARCWriter, separate from combined WARC rollover size:
- check rolloverSize and close previous WARCs when size exceeds
- add timestamp to resource WARC filenames to support rollover, eg.
screenshots-{ts}.warc.gz
- use append mode for all write streams, just in case
- tests: add test for rollover of individual WARCs with 500K size limit
- tests: update screenshot tests to account for WARCs now being named
screenshots-{ts}.warc.gz instead of just screenshots.warc.gz
2024-04-15 13:43:08 -07:00