2020-10-31 13:16:37 -07:00
|
|
|
{
|
|
|
|
|
"name": "browsertrix-crawler",
|
2025-01-30 14:32:30 -08:00
|
|
|
"version": "1.5.0-beta.4",
|
2020-10-31 13:16:37 -07:00
|
|
|
"main": "browsertrix-crawler",
|
2022-10-24 15:30:10 +02:00
|
|
|
"type": "module",
|
2020-10-31 13:16:37 -07:00
|
|
|
"repository": "https://github.com/webrecorder/browsertrix-crawler",
|
|
|
|
|
"author": "Ilya Kreymer <ikreymer@gmail.com>, Webrecorder Software",
|
2022-11-21 09:19:42 -08:00
|
|
|
"license": "AGPL-3.0-or-later",
|
2021-04-30 12:31:14 -07:00
|
|
|
"scripts": {
|
2023-11-09 11:27:11 -08:00
|
|
|
"tsc": "tsc",
|
2024-03-16 17:59:32 -04:00
|
|
|
"format": "prettier src/ --check",
|
|
|
|
|
"format:fix": "prettier src/ --write",
|
|
|
|
|
"lint": "eslint src/",
|
|
|
|
|
"lint:fix": "yarn format:fix && eslint src/ --fix",
|
2023-09-07 13:03:22 -07:00
|
|
|
"test": "yarn node --experimental-vm-modules $(yarn bin jest --bail 1)",
|
|
|
|
|
"prepare": "husky install"
|
2021-04-30 12:31:14 -07:00
|
|
|
},
|
2020-10-31 13:16:37 -07:00
|
|
|
"dependencies": {
|
2024-11-25 18:42:56 -08:00
|
|
|
"@novnc/novnc": "1.4.0",
|
2025-01-06 12:06:40 -08:00
|
|
|
"@webrecorder/wabac": "^2.20.8",
|
Autoclick Support (#729)
Adds support for autoclick behavior:
- Adds new `autoclick` behavior option to `--behaviors`, but not
enabling by default
- Adds support for new exposed function `__bx_addSet` which allows
autoclick behavior to persist state about links that have already been
clicked to avoid duplicates, only used if link has an href
- Adds a new pageFinished flag on the worker state.
- Adds a on('dialog') handler to reject onbeforeunload page navigations,
when in behavior (page not finished), but accept when page is finished -
to allow navigation away only when behaviors are done
- Update to browsertrix-behaviors 0.7.0, which supports autoclick
- Add --clickSelector option to customize elements that will be clicked,
defaulting to `a`.
- Add --linkSelector as alias for --selectLinks for consistency
- Unknown options for --behaviors printed as warnings, instead of hard
exit, for forward compatibility for new behavior types in the future
Fixes #728, also #216, #665, #31
2025-01-16 09:38:11 -08:00
|
|
|
"browsertrix-behaviors": "^0.7.0",
|
Streaming in-place WACZ creation + CDXJ indexing (#673)
Fixes #674
This PR supersedes #505, and instead of using js-wacz for optimized WACZ
creation:
- generates an 'in-place' or 'streaming' WACZ in the crawler, without
having to copy the data again.
- WACZ contents are streamed to remote upload (or to disk) from existing
files on disk
- CDXJ indices per-WARC are first written to 'warc-cdx' directory, then merged using the linux 'sort' command, and compressed to ZipNum if >50K (or always if using --generateCDX)
- All data in the WARCs is written and read only once
- Should result in significant speed / disk usage improvements:
previously WARC was written once, then read again (for CDXJ indexing),
read again (for adding to new WACZ ZIP), written to disk (into new WACZ
ZIP), read again (if upload to remote endpoint). Now, WARCs are written
once, along with the per-WARC CDXJ, the CDXJ only is reread, sorted and merged on-disk, and all
data is read once to either generate WACZ on disk or upload to remote.
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
2024-08-29 13:21:20 -07:00
|
|
|
"client-zip": "^2.4.5",
|
2024-11-08 08:04:41 -08:00
|
|
|
"css-selector-parser": "^3.0.5",
|
2024-06-10 13:11:00 -07:00
|
|
|
"fetch-socks": "^1.3.0",
|
2022-10-24 15:30:10 +02:00
|
|
|
"get-folder-size": "^4.0.0",
|
2023-09-07 13:03:22 -07:00
|
|
|
"husky": "^8.0.3",
|
2023-11-09 11:27:11 -08:00
|
|
|
"ioredis": "^5.3.2",
|
2024-03-22 17:32:42 -07:00
|
|
|
"js-levenshtein": "^1.1.6",
|
Per-Seed Scoping Rules + Crawl Depth (#63)
* scoped seeds:
- support per-seed scoping (include + exclude), allowHash, depth, and sitemap options
- support maxDepth per seed #16
- combine --url, --seed and --urlFile/--seedFile urls into a unified seed list
arg parsing:
- simplify seed file options into --seedFile/--urlFile, move option in help display
- rename --maxDepth -> --depth, supported globally and per seed
- ensure custom parsed params from argParser passed back correctly (behaviors, logging, device emulation)
- update to latest js-yaml
- rename --yamlConfig -> --config
- config: support reading config from stdin if --config set to 'stdin'
* scope: fix typo in 'prefix' scope
* update browsertrix-behaviors to 0.2.2
* tests: add test for passing config via stdin, also adding --excludes via cmdline
* update README:
- latest cli, add docs on config via stdin
- rename --yamlConfig -> --config, consolidate --seedFile/--urlFile, move arg position
- info on scoped seeds
- list current scope types
2021-06-26 13:11:29 -07:00
|
|
|
"js-yaml": "^4.1.0",
|
2023-11-09 11:27:11 -08:00
|
|
|
"minio": "^7.1.3",
|
2023-11-07 21:38:50 -08:00
|
|
|
"p-queue": "^7.3.4",
|
2024-03-22 17:32:42 -07:00
|
|
|
"pixelmatch": "^5.3.0",
|
|
|
|
|
"pngjs": "^7.0.0",
|
Autoclick Support (#729)
Adds support for autoclick behavior:
- Adds new `autoclick` behavior option to `--behaviors`, but not
enabling by default
- Adds support for new exposed function `__bx_addSet` which allows
autoclick behavior to persist state about links that have already been
clicked to avoid duplicates, only used if link has an href
- Adds a new pageFinished flag on the worker state.
- Adds a on('dialog') handler to reject onbeforeunload page navigations,
when in behavior (page not finished), but accept when page is finished -
to allow navigation away only when behaviors are done
- Update to browsertrix-behaviors 0.7.0, which supports autoclick
- Add --clickSelector option to customize elements that will be clicked,
defaulting to `a`.
- Add --linkSelector as alias for --selectLinks for consistency
- Unknown options for --behaviors printed as warnings, instead of hard
exit, for forward compatibility for new behavior types in the future
Fixes #728, also #216, #665, #31
2025-01-16 09:38:11 -08:00
|
|
|
"puppeteer-core": "^24.1.0",
|
SAX-based sitemap parser (#497)
Adds a new SAX-based sitemap parser, inspired by:
https://www.npmjs.com/package/sitemap-stream-parser
Supports:
- recursively parsing sitemap indexes, using p-queue to process N at a
time (currently 5)
- `fromDate` and `toDate` filter dates, to only include URLs between the given
dates, filtering nested sitemap lists included
- async parsing, continue parsing in the background after 100 URLs
- timeout for initial fetch / first 100 URLs set to 30 seconds to avoid
slowing down the crawl
- save/load state integration: mark if sitemaps have already been parsed
in redis, serialize to save state, to avoid reparsing again. (Will
reparse if parsing did not fully finish)
- Aware of `pageLimit`, don't add URLs pass the page limit, interrupt
further parsing when at limit.
- robots.txt `sitemap:` parsing, check URL extension and mime type
- automatic detection of sitemaps for a seed URL if no sitemap url provided - first check robots.txt,
then /sitemap.xml
- tests: test for full sitemap autodetect, sitemap with limit, and sitemap from specific URL.
Fixes #496
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
2024-03-18 19:14:07 -07:00
|
|
|
"sax": "^1.3.0",
|
2023-11-16 16:18:00 -05:00
|
|
|
"sharp": "^0.32.6",
|
2023-11-09 11:27:11 -08:00
|
|
|
"tsc": "^2.0.4",
|
2024-06-13 18:46:05 -07:00
|
|
|
"undici": "^6.18.2",
|
2021-02-17 12:37:07 -05:00
|
|
|
"uuid": "8.3.2",
|
separate fetch api for autofetch bbehavior + additional improvements on partial responses: (#736)
Chromium now interrupts fetch() if abort() is called or page is
navigated, so autofetch behavior using native fetch() is less than
ideal. This PR adds support for __bx_fetch() command for autofetch
behavior (supported in browsertrix-behaviors 0.6.6) to fetch separately
from browser's reguar fetch()
- __bx_fetch() starts a fetch, but does not return content to browser,
doesn't need abort(), unaffected by page navigation, but will still try
to use browser network stack when possible, making it more efficient for
background fetching.
- if network stack fetch fails, fallback to regular node fetch() in the
crawler.
Additional improvements for interrupted fetch:
- don't store truncated media responses, even for 200
- avoid doing duplicate async fetching if response already handled (eg.
fetch handled in multiple contexts)
- fixes #735, where fetch was interrupted, resulted in an empty response
2024-12-31 13:52:12 -08:00
|
|
|
"warcio": "^2.4.3",
|
2021-06-07 17:43:36 -07:00
|
|
|
"ws": "^7.4.4",
|
2023-08-13 15:08:36 -07:00
|
|
|
"yargs": "^17.7.2"
|
2021-02-17 12:37:07 -05:00
|
|
|
},
|
2020-11-01 19:22:53 -08:00
|
|
|
"devDependencies": {
|
2024-03-22 17:32:42 -07:00
|
|
|
"@types/js-levenshtein": "^1.1.3",
|
2023-11-09 11:27:11 -08:00
|
|
|
"@types/js-yaml": "^4.0.8",
|
|
|
|
|
"@types/node": "^20.8.7",
|
2024-03-22 17:32:42 -07:00
|
|
|
"@types/pixelmatch": "^5.2.6",
|
|
|
|
|
"@types/pngjs": "^6.0.4",
|
Streaming in-place WACZ creation + CDXJ indexing (#673)
Fixes #674
This PR supersedes #505, and instead of using js-wacz for optimized WACZ
creation:
- generates an 'in-place' or 'streaming' WACZ in the crawler, without
having to copy the data again.
- WACZ contents are streamed to remote upload (or to disk) from existing
files on disk
- CDXJ indices per-WARC are first written to 'warc-cdx' directory, then merged using the linux 'sort' command, and compressed to ZipNum if >50K (or always if using --generateCDX)
- All data in the WARCs is written and read only once
- Should result in significant speed / disk usage improvements:
previously WARC was written once, then read again (for CDXJ indexing),
read again (for adding to new WACZ ZIP), written to disk (into new WACZ
ZIP), read again (if upload to remote endpoint). Now, WARCs are written
once, along with the per-WARC CDXJ, the CDXJ only is reread, sorted and merged on-disk, and all
data is read once to either generate WACZ on disk or upload to remote.
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
2024-08-29 13:21:20 -07:00
|
|
|
"@types/sax": "^1.2.7",
|
2023-11-09 11:27:11 -08:00
|
|
|
"@types/uuid": "^9.0.6",
|
|
|
|
|
"@types/ws": "^8.5.8",
|
|
|
|
|
"@typescript-eslint/eslint-plugin": "^6.10.0",
|
|
|
|
|
"@typescript-eslint/parser": "^6.10.0",
|
|
|
|
|
"eslint": "^8.53.0",
|
2023-11-09 19:11:11 -05:00
|
|
|
"eslint-config-prettier": "^9.0.0",
|
2021-03-01 19:00:23 -05:00
|
|
|
"eslint-plugin-react": "^7.22.0",
|
2024-06-20 16:35:30 -07:00
|
|
|
"http-server": "^14.1.1",
|
2024-03-22 17:32:42 -07:00
|
|
|
"jest": "^29.7.0",
|
2023-11-09 11:27:11 -08:00
|
|
|
"md5": "^2.3.0",
|
2023-11-09 19:11:11 -05:00
|
|
|
"prettier": "3.0.3",
|
2024-10-16 21:02:04 -07:00
|
|
|
"typescript": "^5.5.4"
|
2022-10-24 15:30:10 +02:00
|
|
|
},
|
|
|
|
|
"jest": {
|
|
|
|
|
"transform": {},
|
|
|
|
|
"testTimeout": 90000
|
Streaming in-place WACZ creation + CDXJ indexing (#673)
Fixes #674
This PR supersedes #505, and instead of using js-wacz for optimized WACZ
creation:
- generates an 'in-place' or 'streaming' WACZ in the crawler, without
having to copy the data again.
- WACZ contents are streamed to remote upload (or to disk) from existing
files on disk
- CDXJ indices per-WARC are first written to 'warc-cdx' directory, then merged using the linux 'sort' command, and compressed to ZipNum if >50K (or always if using --generateCDX)
- All data in the WARCs is written and read only once
- Should result in significant speed / disk usage improvements:
previously WARC was written once, then read again (for CDXJ indexing),
read again (for adding to new WACZ ZIP), written to disk (into new WACZ
ZIP), read again (if upload to remote endpoint). Now, WARCs are written
once, along with the per-WARC CDXJ, the CDXJ only is reread, sorted and merged on-disk, and all
data is read once to either generate WACZ on disk or upload to remote.
---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
2024-08-29 13:21:20 -07:00
|
|
|
},
|
|
|
|
|
"resolutions": {
|
2024-09-05 11:10:31 -07:00
|
|
|
"wrap-ansi": "7.0.0",
|
separate fetch api for autofetch bbehavior + additional improvements on partial responses: (#736)
Chromium now interrupts fetch() if abort() is called or page is
navigated, so autofetch behavior using native fetch() is less than
ideal. This PR adds support for __bx_fetch() command for autofetch
behavior (supported in browsertrix-behaviors 0.6.6) to fetch separately
from browser's reguar fetch()
- __bx_fetch() starts a fetch, but does not return content to browser,
doesn't need abort(), unaffected by page navigation, but will still try
to use browser network stack when possible, making it more efficient for
background fetching.
- if network stack fetch fails, fallback to regular node fetch() in the
crawler.
Additional improvements for interrupted fetch:
- don't store truncated media responses, even for 200
- avoid doing duplicate async fetching if response already handled (eg.
fetch handled in multiple contexts)
- fixes #735, where fetch was interrupted, resulted in an empty response
2024-12-31 13:52:12 -08:00
|
|
|
"warcio": "^2.4.3",
|
2024-11-25 18:42:56 -08:00
|
|
|
"@novnc/novnc": "1.4.0"
|
2020-10-31 13:16:37 -07:00
|
|
|
}
|
|
|
|
|
}
|