Stowage/browsertrix-crawler
2
0
Fork 0
mirror of https://github.com/webrecorder/browsertrix-crawler.git synced 2025-12-08 06:09:48 +00:00
Code Issues Projects Releases Packages Wiki Activity
592 commits 63 branches 154 tags 87 MiB
Commit graph

2 commits

Author SHA1 Message Date
Ilya Kreymer
ca02f09b5d dedup indexing: strip hash prefix from digest, as cdx does not have it 
tests: add index import + dedup crawl to ensure digests match fully
 2025-11-27 22:28:43 -08:00
Ilya Kreymer
0cadf371d0 tests: add dedup-basic.test for simple dedup, ensure number of revisit records === number of response records 2025-11-27 22:28:13 -08:00