Stowage/browsertrix-crawler
2
0
Fork 0
mirror of https://github.com/webrecorder/browsertrix-crawler.git synced 2025-12-08 06:09:48 +00:00
Code Issues Projects Releases Packages Wiki Activity

seed urls list: check for quoted URLs and remove quotes (#883)

Browse source 
- check for urls that are wrapped in quotes, eg. 'https://example.com/'
or "https://example.com/" and trim and remove the quotes before adding seed
- tests: add quoted URL to tests, fix old.webrecorder.net test
- deps: update wabac.js, RWP to latest
- logging: reduce error logging for seed lists, only log once that there are duplicates or page limit is reached
- fix for #882
This commit is contained in:
Ilya Kreymer 2025-09-12 13:34:41 -07:00 • committed by GitHub
parent 705bc0cd9f
commit a2742df328
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 68 additions and 33 deletions
Download patch file Download diff file Expand all files Collapse all files

2
Dockerfile
View file

@ -39,7 +39,7 @@ ADD config/ /app/
ADD html/ /app/html/
ARG RWP_VERSION=2.3.15
ARG RWP_VERSION=2.3.17
ADD https://cdn.jsdelivr.net/npm/replaywebpage@${RWP_VERSION}/ui.js /app/html/rwp/
ADD https://cdn.jsdelivr.net/npm/replaywebpage@${RWP_VERSION}/sw.js /app/html/rwp/
ADD https://cdn.jsdelivr.net/npm/replaywebpage@${RWP_VERSION}/adblock/adblock.gz /app/html/rwp/adblock.gz