browsertrix-crawler/util
Ilya Kreymer 9f541ab011
Support for uploading to S3 (#95)
- support uploading WACZ to s3-compatible storage (via minio client)
- config storage loaded from env vars, enabled when WACZ output is used.
- support pinging either or an http or a redis key-based webhook,
- webhook: include 'completed' bool to indicate if fully completed crawl or partial (eg. interrupted via signal)
- consolidate redis init to redis.js
- support upload filename with custom variables: can interpolate current timestamp (@ts), hostname (@hostname) and user provided id (@crawlId)
- README: add docs for s3 storage, remove unused args
- update to pywb 2.6.2, browsertrix-behaviors 0.2.4

* fix to `limit` option, ensure limit check uses shared state

* bump version to 0.5.0-beta.1
2021-11-23 12:53:30 -08:00
..
argParser.js State Save + Restore State from Config + Redis State + Scope Fix 0.5.0 (#78) 2021-09-28 09:41:16 -07:00
blockrules.js Page Resource Block Rules Avoid Duplicate Handlers + Ignore top-level pages + README update (0.4.4) (#81) 2021-08-17 20:54:18 -07:00
browser.js 0.4.1 Release! (#70) 2021-07-22 14:24:51 -07:00
constants.js Customizable extract selectors + typo fix (0.4.2) (#72) 2021-07-23 18:31:43 -07:00
redis.js Support for uploading to S3 (#95) 2021-11-23 12:53:30 -08:00
screencaster.js 0.4.1 Release! (#70) 2021-07-22 14:24:51 -07:00
seeds.js State Save + Restore State from Config + Redis State + Scope Fix 0.5.0 (#78) 2021-09-28 09:41:16 -07:00
state.js State Save + Restore State from Config + Redis State + Scope Fix 0.5.0 (#78) 2021-09-28 09:41:16 -07:00
storage.js Support for uploading to S3 (#95) 2021-11-23 12:53:30 -08:00
textextract.js Arg Parsing Refactor + Support for YAML Config Support (take 2!) (#59) 2021-06-23 19:45:40 -07:00