5 commits

Author	SHA1	Message	Date
Ilya Kreymer	dd8d2e1ea7	rename 'dedup' -> 'dedupe' for consistency	2025-11-27 22:35:15 -08:00
Ilya Kreymer	6579b2dc95	update to new data model: ... - hashes stored in separate crawl specific entries, h:<crawlid> - wacz files stored in crawl specific list, c:<crawlid>:wacz - hashes committed to 'alldupes' hashset when crawl is complete, crawls added to 'allcrawls' set - store filename, crawlId in related.requires list entries for each wacz	2025-11-27 22:32:52 -08:00
Ilya Kreymer	298b901558	- track source index for each hash, so entry becomes '<source index> <date> <url>' ... - entry for source index can contain the crawl id (or possibly wacz and crawl id) - also store dependent sources in relation.requires in datapackage.json - tests: update tests to check for relation.requires	2025-11-27 22:29:37 -08:00
Ilya Kreymer	ca02f09b5d	dedup indexing: strip hash prefix from digest, as cdx does not have it ... tests: add index import + dedup crawl to ensure digests match fully	2025-11-27 22:28:43 -08:00
Ilya Kreymer	0cadf371d0	tests: add dedup-basic.test for simple dedup, ensure number of revisit records === number of response records	2025-11-27 22:28:13 -08:00