browsertrix-crawler/defaultDriver.js


module.exports = async ({data, page, crawler}) => {
  const {url} = data;
  await crawler.loadPage(page, url);
};
refactor crawler and default driver: - add extensible defaultDriver, wrap crawling functionality in Crawler class - support headless/non-headless, custom driver - support custom collection name for pywb, generate-cdx option - autoplay: add slightly delay for splash loading 2020-11-01 19:22:53 -08:00
			`module.exports = async ({data, page, crawler}) => {`
			`const {url} = data;`
Add scopeType options + option to crawl hashtags + simplify defaultDriver.js (#51) * support hashtag for page-scoped crawls: - allow hashtags for current page, automatically set scope to current w/ different hashtags - also allow hashtags for URLs specified via urlFile - driver: simplify driver, move default driver function to loadPage() - bump version to 0.4.0-beta.0 * add --allowHash option to allow hashtags in URLs, enabled for --spaMode but can be set for crawling as well * graceful shutdown: ensure redis and pywb processes shutdown on exit (for use with singularity, outside of docker) * replace spaMode with more generic --scopeType, a shortcut to setting the scope via regex. scopeType options include: prefix - scope is prefix of current page (default) page - scope is current page + hashtags (spa mode) domain - scope is domain/origin of current page any - scope is any url (default for urlFile) - bump version to 0.4.0-beta.1 2021-05-21 15:37:02 -07:00			`await crawler.loadPage(page, url);`
refactor crawler and default driver: - add extensible defaultDriver, wrap crawling functionality in Crawler class - support headless/non-headless, custom driver - support custom collection name for pywb, generate-cdx option - autoplay: add slightly delay for splash loading 2020-11-01 19:22:53 -08:00			`};`