Stowage/browsertrix-crawler
2
0
Fork 0
mirror of https://github.com/webrecorder/browsertrix-crawler.git synced 2025-10-19 06:23:16 +00:00
Code Issues Projects Releases Packages Wiki Activity
browsertrix-crawler/tests
History
Tessa Walsh f83d0e8f02
Add option to push behavior + behavior script logs to Redis (#805) 
Fixes #804 

- Site-specific behaviors use behaviorScriptCustom log context (via browsertrix-behaviors 0.8.3)
- Add behavior logs to redis if --logBehaviorsToRedis is set, including non-debug behaviors / behaviorsScript context and all behaviorScriptCustom logs
- Noisy logs from built-in behaviors like autoscroll are now logged to
debug in https://github.com/webrecorder/browsertrix-behaviors/pull/92
and so won't be pushed to Redis for newer versions of the crawler.
- Updates browsertrix-behaviors to 0.8.3 and makes some changes to
log format in tests accordingly.

---------

Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
2025-04-03 15:46:10 -07:00
..
custom-behaviors Support loading custom behaviors from URLs and/or filepaths (#707) 2024-11-04 20:30:53 -08:00
fixtures Support custom css selectors for extracting links (#689) 2024-11-08 11:04:41 -05:00
invalid-behaviors detect invalid custom behaviors on load: (#450) 2023-12-13 15:14:53 -05:00
.DS_Store tests text extraction (#30) 2021-03-01 16:00:23 -08:00
adblockrules.test.js tests: reduce logging (#596) 2024-06-26 13:05:13 -07:00
add-exclusion.test.js tests: use old.webrecorder.net for testing (#710) 2024-10-31 13:24:58 -04:00
basic_crawl.test.js Streaming in-place WACZ creation + CDXJ indexing (#673) 2024-08-29 13:21:20 -07:00
blockrules.test.js tests: disable blockrules youtube tests in CI (#698) 2024-10-04 17:37:13 -07:00
brave-query-redir.test.js tests: use old.webrecorder.net for testing (#710) 2024-10-31 13:24:58 -04:00
collection_name.test.js Add Prettier to the repo, and format all the files! (#428) 2023-11-09 16:11:11 -08:00
config_file.test.js Add Prettier to the repo, and format all the files! (#428) 2023-11-09 16:11:11 -08:00
config_stdin.test.js tests: reduce logging (#596) 2024-06-26 13:05:13 -07:00
crawl_overwrite.js Add Prettier to the repo, and format all the files! (#428) 2023-11-09 16:11:11 -08:00
custom-behavior.test.js Add option to push behavior + behavior script logs to Redis (#805) 2025-04-03 15:46:10 -07:00
custom_driver.test.js Support custom css selectors for extracting links (#689) 2024-11-08 11:04:41 -05:00
custom_selector.test.js Validate Autoclick selector, fail crawl if invalid (#800) 2025-03-30 13:48:41 -07:00
dryrun.test.js tests: use old.webrecorder.net for testing (#710) 2024-10-31 13:24:58 -04:00
exclude-redirected.test.js Apply exclusions to redirects (#745) 2025-01-28 11:28:23 -08:00
extra_hops_depth.test.js tests: use old.webrecorder.net for testing (#710) 2024-10-31 13:24:58 -04:00
file_stats.test.js Retry same queue (#757) 2025-02-06 18:48:40 -08:00
http-auth.test.js http auth support per seed (supersedes #566): (#616) 2024-06-20 16:35:30 -07:00
limit_reached.test.js Add more exit codes to detect interruption reason (#764) 2025-02-10 14:00:55 -08:00
log_filtering.test.js Better default crawlId (#806) 2025-04-01 13:40:03 -07:00
mult_url_crawl_with_favicon.test.js tests: use old.webrecorder.net for testing (#710) 2024-10-31 13:24:58 -04:00
multi-instance-crawl.test.js tests: use old.webrecorder.net for testing (#710) 2024-10-31 13:24:58 -04:00
non-html-crawl.test.js Always download PDF + non HTML page cleanup + enterprise policy cleanup (#629) 2024-06-26 09:16:24 -07:00
pageinfo-records.test.js tests: use old.webrecorder.net for testing (#710) 2024-10-31 13:24:58 -04:00
proxy.test.js Strip credentials from proxy address in crawl logs (#778) 2025-02-26 15:23:38 -05:00
qa_compare.test.js tests: update qa test to use awp site 2025-03-21 13:06:53 -07:00
retry-failed.test.js Retry same queue (#757) 2025-02-06 18:48:40 -08:00
rollover-writer.test.js Autoclick Support (#729) 2025-01-16 09:38:11 -08:00
saved-state.test.js tests: use old.webrecorder.net for testing (#710) 2024-10-31 13:24:58 -04:00
scopes.test.js crawler args typing (#680) 2024-09-05 18:10:27 -07:00
screenshot.test.js Fix for --rolloverSize for individual WARCs in 1.x (#542) 2024-04-15 13:43:08 -07:00
seeds.test.js tests: use old.webrecorder.net for testing (#710) 2024-10-31 13:24:58 -04:00
sitemap-parse.test.js stuck page handling: when attempting to restart browser, add more retries (#808) 2025-04-01 16:56:01 -07:00
storage.test.js tests: reduce logging (#596) 2024-06-26 13:05:13 -07:00
text-extract.test.js Add Prettier to the repo, and format all the files! (#428) 2023-11-09 16:11:11 -08:00
upload-wacz.test.js pages redis: include 'depth', 'seed' and 'favIconUrl' in page data added to redis (#749) 2025-01-30 11:18:59 -08:00
url_file_list.test.js Add Prettier to the repo, and format all the files! (#428) 2023-11-09 16:11:11 -08:00
warcinfo.test.js tests: reduce logging (#596) 2024-06-26 13:05:13 -07:00