refactor crawler and default driver:

- add extensible defaultDriver, wrap crawling functionality in Crawler class
- support headless/non-headless, custom driver
- support custom collection name for pywb, generate-cdx option
- autoplay: add slightly delay for splash loading
This commit is contained in:
Ilya Kreymer 2020-11-01 19:22:53 -08:00
parent ded83b52b3
commit 91b8994a08
12 changed files with 1131 additions and 363 deletions

View file

@ -11,5 +11,8 @@
"puppeteer-cluster": "^0.22.0",
"puppeteer-core": "^5.3.1",
"yargs": "^16.0.3"
},
"devDependencies": {
"eslint": "^7.12.1"
}
}