Logging and Behavior Tweaks (#229)

- Ensure page is included in all logging details
- Update logging messages to be a single string, with variables added in the details
- Always wait for all pending wait requests to finish (unless counter <0)
- Don't set puppeteer-cluster timeout (prep for removing puppeeteer-cluster)
- Add behaviorTimeout to running behaviors in crawler, in addition to in behaviors themselves.
- Add logging for behavior start, finish and timeout
- Move writeStats() logging to beginning of each page as well as at the end, to avoid confusion about pending pages.
- For events from frames, use frameUrl along with current page
- deps: bump browsertrix-behaviors to 0.4.2
- version: bump to 0.8.1
This commit is contained in:
Ilya Kreymer 2023-02-23 18:50:22 -08:00 committed by GitHub
parent a4358f4622
commit 5da379cb5f
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
6 changed files with 116 additions and 82 deletions

View file

@ -1,6 +1,6 @@
{
"name": "browsertrix-crawler",
"version": "0.8.0-beta.2",
"version": "0.8.1",
"main": "browsertrix-crawler",
"type": "module",
"repository": "https://github.com/webrecorder/browsertrix-crawler",
@ -13,7 +13,7 @@
"dependencies": {
"@novnc/novnc": "1.4.0-beta",
"abort-controller": "^3.0.0",
"browsertrix-behaviors": "^0.4.1",
"browsertrix-behaviors": "^0.4.2",
"get-folder-size": "^4.0.0",
"ioredis": "^4.27.1",
"js-yaml": "^4.1.0",