Stowage/browsertrix-crawler
2
0
Fork 0
mirror of https://github.com/webrecorder/browsertrix-crawler.git synced 2025-10-19 06:23:16 +00:00
Code Issues Projects Releases Packages Wiki Activity

Add --urlFile param to specify text file with a list of URLs to crawl (#38)

Browse source 
* Resolves #12

* Make --url param optional. Only one of --url of --urlFile should be specified.

* Add ignoreScope option queueUrls() to support adding specific URLs

* add tests for urlFile

* bump version to 0.3.2

Co-authored-by: Emma Dickson <emmadickson@Emmas-MacBook-Pro.local>
This commit is contained in:
Emma Dickson 2021-05-13 01:57:06 -04:00 • committed by GitHub
parent 2db7bc98b1
commit 63376ab6ac
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
10 changed files with 4334 additions and 3633 deletions
Download patch file Download diff file Expand all files Collapse all files

2
package.json
View file

@ -1,6 +1,6 @@
{
"name": "browsertrix-crawler",
"version": "0.3.1",
"version": "0.3.2",
"main": "browsertrix-crawler",
"repository": "https://github.com/webrecorder/browsertrix-crawler",
"author": "Ilya Kreymer <ikreymer@gmail.com>, Webrecorder Software",