* pending request wait:
- instead of waiting for 5s, check redis key 'pywb:{coll}:pending' to see if any pending requests are still pending
- keep checking key until pending requests are at 0
- requires latest pywb 2.6.0+
- should fix#44
* fix test to no longer look for waiting for 5s message
* lint settings and fixes: allow constant in loops, add lint command to script
* chrome: bump default image to chrome:90 image
Support for profiles via a mounted .tar.gz and --profile option + improved docs #18
* support creating profiles via 'create-login-profile' command with options for where to save profile, username/pass and debug screenshot output. support entering username and password (hidden) on command-line if omitted.
* use patched pywb for fix
* bump browsertrix-behaviors to 0.1.0
* README: updates to include better getting started, behaviors and profile reference/examples
* bump version to 0.3.0!
* generates combined WARCs in collection root directory with suffix `_0.warc`, `_1.warc`, etc..
* each combined WARC limited by the size in `--rolloverSize`, if exceeds a new WARC is created, otherwise appended to previous WARC.
* add test for --combineWARC flag
* add improved lint rules
Co-authored-by: Emma Dickson <emmadickson@Emmas-MacBook-Pro.local>
- inject built 'behaviors.js' from browsertrix-behaviors, init with options and run
- remove bgbehaviors
- move textextract to root for now
- add requirements.txt for python dependencies
- remove obsolete --scroll option, to part of the behaviors system
logging:
- configure logging options via --logging param, can include 'stats' (default), 'pywb', 'behaviors', and 'behaviors-debug'
- inject custom logging function for behaviors to call if either behaviors or behaviors-debug is set
- 'behaviors-debug' prints all debug messages from behaviors, while regular 'behaviors' prints main behavior messages (useful for verification)
dockerfile: add 'rebuild' arg to faciliate rebuilding image from specific step
bump to 0.3.0-beta.0
* make --generateWacz, --generateCdx case insensitive with alias option
* fix eslint config and eslint issues
Co-authored-by: Emma Dickson <emmadickson@Emmas-MacBook-Pro.local>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
* linting with eslint
* ci: validate linting and check basic single-page crawl with wacz creation
Co-authored-by: Emma Dickson <emmadickson@Emmas-MacBook-Pro.local>
* Add WACZ creation support, fixes#2
* --generateWACZ flag adds WACZ file (currently named <collection>/<collection>.wacz)
* page list generated in <collection>/pages/pages.jsonl, entry for each page is appended to end of file, includes url and title of page
Co-authored-by: Emma Dickson <emmadickson@Emmas-MacBook-Pro.local>
Co-authored-by: Ilya Kreymer <ikreymer@gmail.com>
- add support for --userAgent to override user agent
- add support for --mobileDevice to use puppeteer device emulation presets
- add support for --userAgentSuffix to append to default user agent (including device userAgent)
bump to 0.1.2
- add extensible defaultDriver, wrap crawling functionality in Crawler class
- support headless/non-headless, custom driver
- support custom collection name for pywb, generate-cdx option
- autoplay: add slightly delay for splash loading
