Stowage/browsertrix-crawler
2
0
Fork 0
mirror of https://github.com/webrecorder/browsertrix-crawler.git synced 2025-10-19 06:23:16 +00:00
Code Issues Projects Releases Packages Wiki Activity

Switch back to Puppeteer from Playwright (#301)

Browse source 
- reduced memory usage, avoids memory leak issues caused by using playwright (see #298) 
- browser: split Browser into Browser and BaseBrowser
- browser: puppeteer-specific functions added to Browser for additional flexibility if need to change again later
- browser: use defaultArgs from playwright
- browser: attempt to recover if initial target is gone
- logging: add debug logging from process.memoryUsage() after every page
- request interception: use priorities for cooperative request interception
- request interception: move to setupPage() to run once per page, enable if any of blockrules, adblockrules or originOverrides are used
- request interception: fix originOverrides enabled check, fix to work with catch-all request interception
- default args: set --waitUntil back to 'load,networkidle2'
- Update README with changes for puppeteer
- tests: fix extra hops depth test to ensure more than one page crawled

---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>
This commit is contained in:
Ilya Kreymer 2023-04-26 15:41:35 -07:00 • committed by GitHub
parent d4e222fab2
commit 71b618fe94
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
12 changed files with 567 additions and 210 deletions
Download patch file Download diff file Expand all files Collapse all files

35
README.md
View file

@ -1,6 +1,6 @@
# Browsertrix Crawler
Browsertrix Crawler is a simplified (Chrome) browser-based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker container. Browsertrix Crawler uses [Playwright](https://github.com/microsoft/playwright) to control one or more browser windows in parallel.
Browsertrix Crawler is a simplified (Chrome) browser-based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker container. Browsertrix Crawler uses [Puppeteer](https://github.com/puppeteer/puppeteer) to control one or more browser windows in parallel.
## Features
@ -14,7 +14,7 @@ Thus far, Browsertrix Crawler supports:
- Screencasting: Ability to watch crawling in real-time (experimental).
- Screenshotting: Ability to take thumbnails, full page screenshots, and/or screenshots of the initial page view.
- Optimized (non-browser) capture of non-HTML resources.
- Extensible Playwright driver script for customizing behavior per crawl or page.
- Extensible Puppeteer driver script for customizing behavior per crawl or page.
- Ability to create and reuse browser profiles interactively or via automated user/password login using an embedded browser.
- Multi-platform support -- prebuilt Docker images available for Intel/AMD and Apple Silicon (M1/M2) CPUs.
@ -69,13 +69,14 @@ Options:
--crawlId, --id A user provided ID for this crawl or
crawl configuration (can also be se
t via CRAWL_ID env var)
[string] [default: "454230b33b8f"]
[string] [default: "97792ef37eaf"]
--newContext Deprecated as of 0.8.0, any values p
assed will be ignored
[string] [default: null]
--waitUntil Playwright page.goto() condition to
wait for before continuing
[default: "load"]
--waitUntil Puppeteer page.goto() condition to w
ait for before continuing, can be mu
ltiple separated by ','
[default: "load,networkidle2"]
--depth The depth of the crawl for all seeds
[number] [default: -1]
--extraHops Number of extra 'hops' to follow, be
@ -150,10 +151,9 @@ Options:
o process.cwd()
[string] [default: "/crawls"]
--mobileDevice Emulate mobile device by name from:
https://github.com/microsoft/playwri
ght/blob/main/packages/playwright-co
re/src/server/deviceDescriptorsSourc
e.json [string]
https://github.com/puppeteer/puppete
er/blob/main/src/common/DeviceDescri
ptors.ts [string]
--userAgent Override user-agent with specified s
tring [string]
--userAgentSuffix Append suffix to existing browser us
@ -240,6 +240,13 @@ Options:
--description, --desc If set, write supplied description i
nto WACZ datapackage.json metadata
[string]
--originOverride if set, will redirect requests from
each origin in key to origin in the
value, eg. --originOverride https://
host:port=http://alt-host:alt-port
[array] [default: []]
--logErrorsToRedis If set, write error messages to redi
s [boolean] [default: false]
--config Path to YAML config file
```
@ -250,9 +257,9 @@ Options:
One of the key nuances of browser-based crawling is determining when a page is finished loading. This can be configured with the `--waitUntil` flag.
The default is `load`, which waits until page load, but for static sites, `--wait-until domcontentloaded` may be used to speed up the crawl (to avoid waiting for ads to load for example). The `--waitUntil networkidle` may make sense for sites where absolutely all requests must be waited until before proceeding.
The default is `load,networkidle2`, which waits until page load and <=2 requests remain, but for static sites, `--wait-until domcontentloaded` may be used to speed up the crawl (to avoid waiting for ads to load for example). `--waitUntil networkidle0` may make sense for sites where absolutely all requests must be waited until before proceeding.
See [page.goto waitUntil options](https://playwright.dev/docs/api/class-page#page-goto-option-wait-until) for more info on the options that can be used with this flag from the Playwright docs.
See [page.goto waitUntil options](https://pptr.dev/api/puppeteer.page.goto#remarks) for more info on the options that can be used with this flag from the Puppeteer docs.
The `--pageLoadTimeout`/`--timeout` option sets the timeout in seconds for page load, defaulting to 90 seconds. Behaviors will run on the page once either the page load condition or the page load timeout is met, whichever happens first.
@ -543,11 +550,11 @@ The webhook URL can be an HTTP URL which receives a JSON POST request OR a Redis
</details>
### Configuring Chromium / Playwright / pywb
### Configuring Chromium / Puppeteer / pywb
There is a few environment variables you can set to configure chromium and pywb:
- CHROME_FLAGS will be split by spaces and passed to Chromium (via `args` in Playwright). Note that setting some options is not supported such as `--proxy-server` since they are set by browsertrix itself.
- CHROME_FLAGS will be split by spaces and passed to Chromium (via `args` in Puppeteer). Note that setting some options is not supported such as `--proxy-server` since they are set by browsertrix itself.
- SOCKS_HOST and SOCKS_PORT are read by pywb to proxy upstream traffic
Here's some examples use cases:

66
crawler.js
View file

@ -355,6 +355,24 @@ export class Crawler {
async setupPage({page, cdp, workerid}) {
await this.browser.setupPage({page, cdp});
if ((this.adBlockRules && this.params.blockAds) ||
this.blockRules || this.originOverride) {
await page.setRequestInterception(true);
if (this.adBlockRules && this.params.blockAds) {
await this.adBlockRules.initPage(this.browser, page);
}
if (this.blockRules) {
await this.blockRules.initPage(this.browser, page);
}
if (this.originOverride) {
await this.originOverride.initPage(this.browser, page);
}
}
if (this.params.logging.includes("jserrors")) {
page.on("console", (msg) => {
if (msg.type() === "error") {
@ -374,7 +392,7 @@ export class Crawler {
if (this.params.behaviorOpts) {
await page.exposeFunction(BEHAVIOR_LOG_FUNC, (logdata) => this._behaviorLog(logdata, page.url(), workerid));
await page.addInitScript(behaviors + `;\nself.__bx_behaviors.init(${this.params.behaviorOpts});`);
await this.browser.addInitScript(page, behaviors + `;\nself.__bx_behaviors.init(${this.params.behaviorOpts});`);
}
}
@ -404,7 +422,7 @@ export class Crawler {
logger.debug("Skipping screenshots for non-HTML page", logDetails);
}
const archiveDir = path.join(this.collDir, "archive");
const screenshots = new Screenshots({page, url, directory: archiveDir});
const screenshots = new Screenshots({browser: this.browser, page, url, directory: archiveDir});
if (this.params.screenshot.includes("view")) {
await screenshots.take();
}
@ -430,7 +448,7 @@ export class Crawler {
logger.info("Skipping behaviors for slow page", logDetails, "behavior");
} else {
const res = await timedRun(
this.runBehaviors(page, data.filteredFrames, logDetails),
this.runBehaviors(page, cdp, data.filteredFrames, logDetails),
this.params.behaviorTimeout,
"Behaviors timed out",
logDetails,
@ -495,16 +513,14 @@ export class Crawler {
}
}
async runBehaviors(page, frames, logDetails) {
async runBehaviors(page, cdp, frames, logDetails) {
try {
frames = frames || page.frames();
const context = page.context();
logger.info("Running behaviors", {frames: frames.length, frameUrls: frames.map(frame => frame.url()), ...logDetails}, "behavior");
return await Promise.allSettled(
frames.map(frame => this.browser.evaluateWithCLI(context, frame, "self.__bx_behaviors.run();", logDetails, "behavior"))
frames.map(frame => this.browser.evaluateWithCLI(page, frame, cdp, "self.__bx_behaviors.run();", logDetails, "behavior"))
);
} catch (e) {
@ -711,7 +727,7 @@ export class Crawler {
this.screencaster = this.initScreenCaster();
if (this.params.originOverride) {
if (this.params.originOverride.length) {
this.originOverride = new OriginOverride(this.params.originOverride);
}
@ -905,6 +921,14 @@ export class Crawler {
});
}
logMemory() {
const memUsage = process.memoryUsage();
const { heapUsed, heapTotal } = memUsage;
this.maxHeapUsed = Math.max(this.maxHeapUsed || 0, heapUsed);
this.maxHeapTotal = Math.max(this.maxHeapTotal || 0, heapTotal);
logger.debug("Memory", {maxHeapUsed: this.maxHeapUsed, maxHeapTotal: this.maxHeapTotal, ...memUsage}, "memory");
}
async writeStats(toFile=false) {
if (!this.params.logging.includes("stats")) {
return;
@ -926,6 +950,7 @@ export class Crawler {
};
logger.info("Crawl statistics", stats, "crawlStatus");
this.logMemory();
if (toFile && this.params.statsFilename) {
try {
@ -965,18 +990,6 @@ export class Crawler {
}
}
if (this.adBlockRules && this.params.blockAds) {
await this.adBlockRules.initPage(page);
}
if (this.blockRules) {
await this.blockRules.initPage(page);
}
if (this.originOverride) {
await this.originOverride.initPage(page);
}
let ignoreAbort = false;
// Detect if ERR_ABORTED is actually caused by trying to load a non-page (eg. downloadable PDF),
@ -998,7 +1011,7 @@ export class Crawler {
try {
const resp = await page.goto(url, gotoOpts);
const contentType = await resp.headerValue("content-type");
const contentType = await this.browser.responseHeader(resp, "content-type");
isHTMLPage = this.isHTMLContentType(contentType);
@ -1068,7 +1081,7 @@ export class Crawler {
await sleep(0.5);
try {
await page.waitForLoadState("networkidle", {timeout: this.params.netIdleWait * 1000});
await this.browser.waitForNetworkIdle(page, {timeout: this.params.netIdleWait * 1000});
} catch (e) {
logger.debug("waitForNetworkIdle timed out, ignoring", details);
// ignore, continue
@ -1095,7 +1108,7 @@ export class Crawler {
try {
const linkResults = await Promise.allSettled(
frames.map(frame => timedRun(
frame.evaluate(loadFunc, {selector: selector, extract: extract}),
frame.evaluate(loadFunc, {selector, extract}),
PAGE_OP_TIMEOUT_SECS,
"Link extraction timed out",
logDetails,
@ -1152,9 +1165,10 @@ export class Crawler {
try {
logger.debug("Check CF Blocking", logDetails);
const cloudflare = page.locator("div.cf-browser-verification.cf-im-under-attack");
while (await cloudflare.waitFor({timeout: PAGE_OP_TIMEOUT_SECS})) {
while (await timedRun(
page.$("div.cf-browser-verification.cf-im-under-attack"),
PAGE_OP_TIMEOUT_SECS
)) {
logger.debug("Cloudflare Check Detected, waiting for reload...", logDetails);
await sleep(5.5);
}

17
create-login-profile.js
View file

@ -158,10 +158,8 @@ async function main() {
const browser = new Browser();
const profileDir = await browser.loadProfile(params.profile);
await browser.launch({
dataDir: profileDir,
profileUrl: params.profile,
headless: params.headless,
signals: true,
chromeOptions: {
@ -191,18 +189,17 @@ async function main() {
params.password = await promptInput("Enter password: ", true);
}
const { page, cdp } = await browser.getFirstPageWithCDP();
const { page, cdp } = await browser.newWindowPageWithCDP();
const waitUntil = "load";
const waitUntil = "load";
//await page.setCacheEnabled(false);
await cdp.send("Network.setCacheDisabled", {cacheDisabled: true});
await page.setCacheEnabled(false);
if (!params.automated) {
await browser.setupPage({page, cdp});
// for testing, inject browsertrix-behaviors
await page.addInitScript(behaviors + ";\nself.__bx_behaviors.init();");
await browser.addInitScript(page, behaviors + ";\nself.__bx_behaviors.init();");
}
logger.info(`Loading page: ${params.url}`);
@ -384,7 +381,7 @@ class InteractiveBrowser {
return;
}
const cookies = await this.browser.context.cookies(url);
const cookies = await this.browser.getCookies(this.page, url);
for (const cookie of cookies) {
cookie.expires = (new Date().getTime() / 1000) + this.params.cookieDays * 86400;
delete cookie.size;
@ -396,7 +393,7 @@ class InteractiveBrowser {
cookie.url = url;
}
}
await this.browser.context.addCookies(cookies);
await this.browser.setCookies(this.page, cookies);
} catch (e) {
logger.error("Save Cookie Error: ", e);
}

2
package.json
View file

@ -17,7 +17,7 @@
"ioredis": "^4.27.1",
"js-yaml": "^4.1.0",
"minio": "7.0.26",
"playwright-core": "^1.31.2",
"puppeteer-core": "^19.11.1",
"sitemapper": "^3.1.2",
"uuid": "8.3.2",
"warcio": "^1.6.0",

8
tests/extra_hops_depth.test.js
View file

@ -16,7 +16,8 @@ test("check that URLs are crawled 2 extra hops beyond depth", async () => {
console.log(error);
}
const crawled_pages = fs.readFileSync("test-crawls/collections/extra-hops-beyond/pages/pages.jsonl", "utf8");
const crawledPages = fs.readFileSync("test-crawls/collections/extra-hops-beyond/pages/pages.jsonl", "utf8");
const crawledPagesArray = crawledPages.trim().split("\n");
const expectedPages = [
"https://webrecorder.net/",
@ -28,7 +29,10 @@ test("check that URLs are crawled 2 extra hops beyond depth", async () => {
"https://webrecorder.net/faq",
];
for (const page of crawled_pages.trim().split("\n")) {
// first line is the header, not page, so adding -1
expect(expectedPages.length).toEqual(crawledPagesArray.length - 1);
for (const page of crawledPagesArray) {
const url = JSON.parse(page).url;
if (!url) {
continue;

30
util/argParser.js
View file

@ -3,7 +3,7 @@ import fs from "fs";
import os from "os";
import yaml from "js-yaml";
import { devices } from "playwright-core";
import { KnownDevices as devices } from "puppeteer-core";
import yargs from "yargs";
import { hideBin } from "yargs/helpers";
@ -52,8 +52,8 @@ class ArgParser {
},
"waitUntil": {
describe: "Playwright page.goto() condition to wait for before continuing",
default: "load",
describe: "Puppeteer page.goto() condition to wait for before continuing, can be multiple separated by ','",
default: "load,networkidle2",
},
"depth": {
@ -208,7 +208,7 @@ class ArgParser {
},
"mobileDevice": {
describe: "Emulate mobile device by name from: https://github.com/microsoft/playwright/blob/main/packages/playwright-core/src/server/deviceDescriptorsSource.json",
describe: "Emulate mobile device by name from: https://github.com/puppeteer/puppeteer/blob/main/src/common/DeviceDescriptors.ts",
type: "string",
},
@ -369,10 +369,10 @@ class ArgParser {
},
"logErrorsToRedis": {
descripe: "If set, write error messages to redis",
describe: "If set, write error messages to redis",
type: "boolean",
default: false,
}
},
};
}
@ -410,10 +410,17 @@ class ArgParser {
logger.fatal(`\n${argv.collection} is an invalid collection name. Please supply a collection name only using alphanumeric characters and the following characters [_ - ]\n`);
}
// waitUntil condition must be one of WAIT_UNTIL_OPTS: load, domcontentloaded, networkidle
// (see: https://playwright.dev/docs/api/class-page#page-goto-option-wait-until)
if (!WAIT_UNTIL_OPTS.includes(argv.waitUntil)) {
logger.fatal("Invalid waitUntil option, must be one of: " + WAIT_UNTIL_OPTS.join(","));
// waitUntil condition must be: load, domcontentloaded, networkidle0, networkidle2
// can be multiple separate by comma
// (see: https://github.com/puppeteer/puppeteer/blob/main/docs/api.md#pagegotourl-options)
if (typeof argv.waitUntil != "object"){
argv.waitUntil = argv.waitUntil.split(",");
}
for (const opt of argv.waitUntil) {
if (!WAIT_UNTIL_OPTS.includes(opt)) {
logger.fatal("Invalid waitUntil option, must be one of: " + WAIT_UNTIL_OPTS.join(","));
}
}
// validate screenshot options
@ -453,9 +460,6 @@ class ArgParser {
if (!argv.emulateDevice) {
logger.fatal("Unknown device: " + argv.mobileDevice);
}
if (argv.emulateDevice.defaultBrowserType !== "chromium") {
logger.fatal(`Device Browser: ${argv.emulateDevice.defaultBrowserType}\r\nSorry, only Chromium-based devices are supported at this time`);
}
} else {
argv.emulateDevice = {viewport: null};
}

43
util/blockrules.js
View file

@ -68,29 +68,19 @@ export class BlockRules
}
}
async initPage(page) {
if (!this.rules.length) {
return;
}
if (page._btrix_interceptionAdded) {
return true;
}
page._btrix_interceptionAdded = true;
await page.route("**/*", (route) => {
async initPage(browser, page) {
const onRequest = async (request) => {
const logDetails = {page: page.url()};
try {
this.handleRequest(route, logDetails);
await this.handleRequest(request, logDetails);
} catch (e) {
logger.warn("Error handling request", {...errJSON(e), ...logDetails}, "blocking");
}
});
};
await browser.interceptRequest(page, onRequest);
}
async handleRequest(route, logDetails) {
const request = route.request();
async handleRequest(request, logDetails) {
const url = request.url();
let blockState;
@ -99,9 +89,9 @@ export class BlockRules
blockState = await this.shouldBlock(request, url, logDetails);
if (blockState === BlockState.ALLOW) {
await route.continue();
await request.continue({}, 1);
} else {
await route.abort("blockedbyclient");
await request.abort("blockedbyclient", 1);
}
} catch (e) {
@ -236,23 +226,6 @@ export class AdBlockRules extends BlockRules
this.adhosts = JSON.parse(fs.readFileSync(new URL(adhostsFilePath, import.meta.url)));
}
async initPage(page) {
if (page._btrix_adInterceptionAdded) {
return true;
}
page._btrix_adInterceptionAdded = true;
await page.route("**/*", (route) => {
const logDetails = {page: page.url()};
try {
this.handleRequest(route, logDetails);
} catch (e) {
logger.warn("Error handling request", {...errJSON(e), ...logDetails}, "blocking");
}
});
}
isAdUrl(url) {
const fragments = url.split("/");
const domain = fragments.length > 2 ? fragments[2] : null;

249
util/browser.js
View file

@ -9,36 +9,32 @@ import path from "path";
import { logger } from "./logger.js";
import { initStorage } from "./storage.js";
import { chromium } from "playwright-core";
import puppeteer from "puppeteer-core";
// ==================================================================
export class Browser
export class BaseBrowser
{
constructor() {
this.context = null;
this.firstPage = null;
this.firstCDP = null;
this.profileDir = fs.mkdtempSync(path.join(os.tmpdir(), "profile-"));
this.customProfile = false;
this.emulateDevice = null;
}
async launch({profileUrl, chromeOptions, signals = false, headless = false, emulateDevice = {viewport: null}} = {}) {
if (this.context) {
logger.warn("Context already inited", {}, "context");
return this.context;
async launch({profileUrl, chromeOptions, signals = false, headless = false, emulateDevice = {}} = {}) {
if (this.isLaunched()) {
return;
}
if (profileUrl) {
this.customProfile = await this.loadProfile(profileUrl);
}
this.emulateDevice = emulateDevice;
const args = this.chromeArgs(chromeOptions);
const launchOpts = {
...emulateDevice,
args,
headless,
executablePath: this.getBrowserExe(),
@ -47,30 +43,17 @@ export class Browser
handleSIGHUP: signals,
handleSIGINT: signals,
handleSIGTERM: signals,
serviceWorkers: "allow"
defaultViewport: null,
waitForInitialPage: false,
userDataDir: this.profileDir
};
this.context = await chromium.launchPersistentContext(this.profileDir, launchOpts);
if (this.context.pages()) {
this.firstPage = this.context.pages()[0];
} else {
this.firstPage = await this.context.newPage();
}
this.firstCDP = await this.context.newCDPSession(this.firstPage);
return this.context;
}
async close() {
if (this.context) {
await this.context.close();
this.context = null;
}
await this._init(launchOpts);
}
async setupPage({page, cdp}) {
await page.addInitScript("Object.defineProperty(navigator, \"webdriver\", {value: false});");
await this.addInitScript(page, "Object.defineProperty(navigator, \"webdriver\", {value: false});");
if (this.customProfile) {
logger.info("Disabling Service Workers for profile", {}, "browser");
@ -79,38 +62,6 @@ export class Browser
}
}
async getFirstPageWithCDP() {
return {page: this.firstPage, cdp: this.firstCDP};
}
numPages() {
return this.context ? this.context.pages().length : 0;
}
async newWindowPageWithCDP() {
// unique url to detect new pages
const startPage = "about:blank?_browsertrix" + Math.random().toString(36).slice(2);
const p = new Promise((resolve) => {
const listener = (page) => {
if (page.url() === startPage) {
resolve(page);
this.context.removeListener("page", listener);
}
};
this.context.on("page", listener);
});
await this.firstCDP.send("Target.createTarget", {url: startPage, newWindow: true});
const page = await p;
const cdp = await this.context.newCDPSession(page);
return {page, cdp};
}
async loadProfile(profileFilename) {
const targetFilename = "/tmp/profile.tar.gz";
@ -157,6 +108,7 @@ export class Browser
chromeArgs({proxy=true, userAgent=null, extraArgs=[]} = {}) {
// Chrome Flags, including proxy server
const args = [
...defaultArgs,
...(process.env.CHROME_FLAGS ?? "").split(" ").filter(Boolean),
//"--no-xshm", // needed for Chrome >80 (check if puppeteer adds automatically)
"--no-sandbox",
@ -200,21 +152,19 @@ export class Browser
return null;
}
async evaluateWithCLI(context, frame, funcString, logData, contextName) {
async evaluateWithCLI_(cdp, frame, cdpContextId, funcString, logData, contextName) {
let details = {frameUrl: frame.url(), ...logData};
logger.info("Run Script Started", details, contextName);
const cdp = await context.newCDPSession(frame);
// from puppeteer _evaluateInternal() but with includeCommandLineAPI: true
//const contextId = context._contextId;
const expression = funcString + "\n//# sourceURL=__playwright_evaluation_script__";
const expression = funcString + "\n//# sourceURL=__evaluation_script__";
const { exceptionDetails, result } = await cdp
.send("Runtime.evaluate", {
expression,
//contextId,
contextId: cdpContextId,
returnByValue: true,
awaitPromise: true,
userGesture: true,
@ -230,13 +180,164 @@ export class Browser
logger.info("Run Script Finished", details, contextName);
}
try {
await cdp.detach();
} catch (e) {
logger.warn("Detach failed", details, contextName);
}
return result.value;
}
}
// ==================================================================
export class Browser extends BaseBrowser
{
constructor() {
super();
this.browser = null;
this.firstCDP = null;
}
isLaunched() {
if (this.browser) {
logger.warn("Context already inited", {}, "browser");
return true;
}
return false;
}
async close() {
if (this.browser) {
await this.browser.close();
this.browser = null;
}
}
addInitScript(page, script) {
return page.evaluateOnNewDocument(script);
}
async _init(launchOpts) {
this.browser = await puppeteer.launch(launchOpts);
const target = this.browser.target();
this.firstCDP = await target.createCDPSession();
}
numPages() {
return this.browser ? this.browser.pages().length : 0;
}
async newWindowPageWithCDP() {
// unique url to detect new pages
const startPage = "about:blank?_browsertrix" + Math.random().toString(36).slice(2);
const p = new Promise((resolve) => {
const listener = (target) => {
if (target.url() === startPage) {
resolve(target);
this.browser.removeListener("targetcreated", listener);
}
};
this.browser.on("targetcreated", listener);
});
try {
await this.firstCDP.send("Target.createTarget", {url: startPage, newWindow: true});
} catch (e) {
const target = this.browser.target();
this.firstCDP = await target.createCDPSession();
await this.firstCDP.send("Target.createTarget", {url: startPage, newWindow: true});
}
const target = await p;
const page = await target.page();
const device = this.emulateDevice;
if (device) {
if (device.viewport && device.userAgent) {
await page.emulate(device);
} else if (device.userAgent) {
await page.setUserAgent(device.userAgent);
}
}
const cdp = await target.createCDPSession();
return {page, cdp};
}
async responseHeader(resp, header) {
return await resp.headers()[header];
}
async evaluateWithCLI(_, frame, cdp, funcString, logData, contextName) {
const context = await frame.executionContext();
cdp = context._client;
const cdpContextId = context._contextId;
return await this.evaluateWithCLI_(cdp, frame, cdpContextId, funcString, logData, contextName);
}
interceptRequest(page, callback) {
page.on("request", callback);
}
async waitForNetworkIdle(page, params) {
return await page.waitForNetworkIdle(params);
}
async setViewport(page, params) {
await page.setViewport(params);
}
async getCookies(page) {
return await page.cookies();
}
async setCookies(page, cookies) {
return await page.setCookie(...cookies);
}
}
// ==================================================================
// Default Chromium args from playwright
export const defaultArgs = [
"--disable-field-trial-config", // https://source.chromium.org/chromium/chromium/src/+/main:testing/variations/README.md
"--disable-background-networking",
"--enable-features=NetworkService,NetworkServiceInProcess",
"--disable-background-timer-throttling",
"--disable-backgrounding-occluded-windows",
"--disable-back-forward-cache", // Avoids surprises like main request not being intercepted during page.goBack().
"--disable-breakpad",
"--disable-client-side-phishing-detection",
"--disable-component-extensions-with-background-pages",
"--disable-component-update", // Avoids unneeded network activity after startup.
"--no-default-browser-check",
"--disable-default-apps",
"--disable-dev-shm-usage",
"--disable-extensions",
// AvoidUnnecessaryBeforeUnloadCheckSync - https://github.com/microsoft/playwright/issues/14047
// Translate - https://github.com/microsoft/playwright/issues/16126
"--disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync,Translate",
"--allow-pre-commit-input",
"--disable-hang-monitor",
"--disable-ipc-flooding-protection",
"--disable-popup-blocking",
"--disable-prompt-on-repost",
"--disable-renderer-backgrounding",
"--disable-sync",
"--force-color-profile=srgb",
"--metrics-recording-only",
"--no-first-run",
"--enable-automation",
"--password-store=basic",
"--use-mock-keychain",
// See https://chromium-review.googlesource.com/c/chromium/src/+/2436773
"--no-service-autorun",
"--export-tagged-pdf"
];

2
util/constants.js
View file

@ -1,6 +1,6 @@
export const HTML_TYPES = ["text/html", "application/xhtml", "application/xhtml+xml"];
export const WAIT_UNTIL_OPTS = ["load", "domcontentloaded", "networkidle"];
export const WAIT_UNTIL_OPTS = ["load", "domcontentloaded", "networkidle0", "networkidle2"];
export const BEHAVIOR_LOG_FUNC = "__bx_log";
export const MAX_DEPTH = 1000000;

56
util/originoverride.js
View file

@ -12,32 +12,40 @@ export class OriginOverride
});
}
initPage(page) {
for (const {orig, dest} of this.originOverride) {
const logDetails = {page: page.url(), orig, dest};
async initPage(browser, page) {
const onRequest = async (request) => {
try {
const url = request.url();
logger.debug(`Adding override ${orig} => ${dest}`);
let newUrl = null;
page.route(orig + "/**", async (route) => {
try {
const request = route.request();
const url = request.url();
const newUrl = dest + url.slice(orig.length);
const resp = await fetch(newUrl, {headers: request.headers()});
const body = Buffer.from(await resp.arrayBuffer());
const headers = Object.fromEntries(resp.headers);
const status = resp.status;
logger.debug("Origin overridden", {orig: url, dest: newUrl, status, body: body.length}, "originoverride");
route.fulfill({body, headers, status});
} catch (e) {
logger.warn("Error overriding origin", {...errJSON(e), ...logDetails}, "originoverride");
for (const {orig, dest} of this.originOverride) {
if (url.startsWith(orig)) {
newUrl = dest + url.slice(orig.length);
break;
}
}
});
}
if (!newUrl) {
request.continue({}, -1);
return;
}
const resp = await fetch(newUrl, {headers: request.headers()});
const body = Buffer.from(await resp.arrayBuffer());
const headers = Object.fromEntries(resp.headers);
const status = resp.status;
logger.debug("Origin overridden", {orig: url, dest: newUrl, status, body: body.length}, "originoverride");
request.respond({body, headers, status}, -1);
} catch (e) {
logger.warn("Error overriding origin", {...errJSON(e), url: page.url()}, "originoverride");
request.continue({}, -1);
}
};
await browser.interceptRequest(page, onRequest);
}
}

5
util/screenshots.js
View file

@ -28,7 +28,8 @@ export const screenshotTypes = {
export class Screenshots {
constructor({page, url, date, directory}) {
constructor({browser, page, url, date, directory}) {
this.browser = browser;
this.page = page;
this.url = url;
this.directory = directory;
@ -39,7 +40,7 @@ export class Screenshots {
async take(screenshotType="view") {
try {
if (screenshotType !== "fullPage") {
await this.page.setViewportSize({width: 1920, height: 1080});
await this.browser.setViewport(this.page, {width: 1920, height: 1080});
}
const options = screenshotTypes[screenshotType];
const screenshotBuffer = await this.page.screenshot(options);

264
yarn.lock
View file

@ -620,6 +620,20 @@
resolved "https://registry.yarnpkg.com/@novnc/novnc/-/novnc-1.4.0.tgz#68adae81a741624142b518323441e852c1f34281"
integrity sha512-kW6ALMc5BuH08e/ond/I1naYcfjc19JYMN1EdtmgjjjzPGCjW8fMtVM3MwM6q7YLRjPlQ3orEvoKMgSS7RkEVQ==
"@puppeteer/browsers@0.5.0":
version "0.5.0"
resolved "https://registry.yarnpkg.com/@puppeteer/browsers/-/browsers-0.5.0.tgz#1a1ee454b84a986b937ca2d93146f25a3fe8b670"
integrity sha512-Uw6oB7VvmPRLE4iKsjuOh8zgDabhNX67dzo8U/BB0f9527qx+4eeUs+korU98OhG5C4ubg7ufBgVi63XYwS6TQ==
dependencies:
debug "4.3.4"
extract-zip "2.0.1"
https-proxy-agent "5.0.1"
progress "2.0.3"
proxy-from-env "1.1.0"
tar-fs "2.1.1"
unbzip2-stream "1.4.3"
yargs "17.7.1"
"@sinclair/typebox@^0.24.1":
version "0.24.50"
resolved "https://registry.yarnpkg.com/@sinclair/typebox/-/typebox-0.24.50.tgz#35ee4db4ab8f3a8ff56490c51f92445d2776451e"
@ -766,6 +780,13 @@
dependencies:
"@types/yargs-parser" "*"
"@types/yauzl@^2.9.1":
version "2.10.0"
resolved "https://registry.yarnpkg.com/@types/yauzl/-/yauzl-2.10.0.tgz#b3248295276cf8c6f153ebe6a9aba0c988cb2599"
integrity sha512-Cn6WYCm0tXv8p6k+A8PvbDG763EDpBoTzHdA+Q/MF6H3sapGjCm9NzoaJncJS9tUKSuCoDs9XHxYYsQDgxR6kw==
dependencies:
"@types/node" "*"
"@zxing/text-encoding@0.9.0":
version "0.9.0"
resolved "https://registry.yarnpkg.com/@zxing/text-encoding/-/text-encoding-0.9.0.tgz#fb50ffabc6c7c66a0c96b4c03e3d9be74864b70b"
@ -781,6 +802,13 @@ acorn@^7.4.0:
resolved "https://registry.yarnpkg.com/acorn/-/acorn-7.4.1.tgz#feaed255973d2e77555b83dbc08851a6c63520fa"
integrity sha512-nQyp0o1/mNdbTO1PO6kHkwSrmgZ0MT/jCCpNiwbUjGoRN4dlBhqJtoQuCnEOKzgTVwg0ZWiCoQy6SxMebQVh8A==
agent-base@6:
version "6.0.2"
resolved "https://registry.yarnpkg.com/agent-base/-/agent-base-6.0.2.tgz#49fff58577cfee3f37176feab4c22e00f86d7f77"
integrity sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ==
dependencies:
debug "4"
ajv@^6.10.0, ajv@^6.12.4:
version "6.12.6"
resolved "https://registry.yarnpkg.com/ajv/-/ajv-6.12.6.tgz#baf5a62e802b07d977034586f8c3baf5adf26df4"
@ -973,6 +1001,20 @@ balanced-match@^1.0.0:
resolved "https://registry.yarnpkg.com/balanced-match/-/balanced-match-1.0.2.tgz#e83e3a7e3f300b34cb9d87f615fa0cbf357690ee"
integrity sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==
base64-js@^1.3.1:
version "1.5.1"
resolved "https://registry.yarnpkg.com/base64-js/-/base64-js-1.5.1.tgz#1b1b440160a5bf7ad40b650f095963481903930a"
integrity sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==
bl@^4.0.3:
version "4.1.0"
resolved "https://registry.yarnpkg.com/bl/-/bl-4.1.0.tgz#451535264182bec2fbbc83a62ab98cf11d9f7b3a"
integrity sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==
dependencies:
buffer "^5.5.0"
inherits "^2.0.4"
readable-stream "^3.4.0"
block-stream2@^2.0.0:
version "2.1.0"
resolved "https://registry.yarnpkg.com/block-stream2/-/block-stream2-2.1.0.tgz#ac0c5ef4298b3857796e05be8ebed72196fa054b"
@ -1091,6 +1133,11 @@ bser@2.1.1:
dependencies:
node-int64 "^0.4.0"
buffer-crc32@~0.2.3:
version "0.2.13"
resolved "https://registry.yarnpkg.com/buffer-crc32/-/buffer-crc32-0.2.13.tgz#0d333e3f00eac50aa1454abd30ef8c2a5d9a7242"
integrity sha512-VO9Ht/+p3SN7SKWqcrgEzjGbRSJYTx+Q1pTQC0wrWqHx0vpJraQ6GtHx8tvcg1rlK1byhU5gccxgOgj7B0TDkQ==
buffer-from@^1.0.0:
version "1.1.1"
resolved "https://registry.yarnpkg.com/buffer-from/-/buffer-from-1.1.1.tgz#32713bc028f75c02fdb710d7c7bcec1f2c6070ef"
@ -1101,6 +1148,14 @@ buffer-xor@^1.0.3:
resolved "https://registry.yarnpkg.com/buffer-xor/-/buffer-xor-1.0.3.tgz#26e61ed1422fb70dd42e6e36729ed51d855fe8d9"
integrity sha1-JuYe0UIvtw3ULm42cp7VHYVf6Nk=
buffer@^5.2.1, buffer@^5.5.0:
version "5.7.1"
resolved "https://registry.yarnpkg.com/buffer/-/buffer-5.7.1.tgz#ba62e7c13133053582197160851a8f648e99eed0"
integrity sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==
dependencies:
base64-js "^1.3.1"
ieee754 "^1.1.13"
cacheable-lookup@^5.0.3:
version "5.0.4"
resolved "https://registry.yarnpkg.com/cacheable-lookup/-/cacheable-lookup-5.0.4.tgz#5a6b865b2c44357be3d5ebc2a467b032719a7005"
@ -1174,6 +1229,18 @@ charenc@0.0.2:
resolved "https://registry.yarnpkg.com/charenc/-/charenc-0.0.2.tgz#c0a1d2f3a7092e03774bfa83f14c0fc5790a8667"
integrity sha1-wKHS86cJLgN3S/qD8UwPxXkKhmc=
chownr@^1.1.1:
version "1.1.4"
resolved "https://registry.yarnpkg.com/chownr/-/chownr-1.1.4.tgz#6fc9d7b42d32a583596337666e7d08084da2cc6b"
integrity sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==
chromium-bidi@0.4.7:
version "0.4.7"
resolved "https://registry.yarnpkg.com/chromium-bidi/-/chromium-bidi-0.4.7.tgz#4c022c2b0fb1d1c9b571fadf373042160e71d236"
integrity sha512-6+mJuFXwTMU6I3vYLs6IL8A1DyQTPjCfIL971X0aMPVGRbGnNfl6i6Cl0NMbxi2bRYLGESt9T2ZIMRM5PAEcIQ==
dependencies:
mitt "3.0.0"
ci-info@^3.2.0:
version "3.5.0"
resolved "https://registry.yarnpkg.com/ci-info/-/ci-info-3.5.0.tgz#bfac2a29263de4c829d806b1ab478e35091e171f"
@ -1308,6 +1375,13 @@ create-hmac@^1.1.0, create-hmac@^1.1.4, create-hmac@^1.1.7:
safe-buffer "^5.0.1"
sha.js "^2.4.8"
cross-fetch@3.1.5:
version "3.1.5"
resolved "https://registry.yarnpkg.com/cross-fetch/-/cross-fetch-3.1.5.tgz#e1389f44d9e7ba767907f7af8454787952ab534f"
integrity sha512-lvb1SBsI0Z7GDwmuid+mU3kWVBwTVUbe7S0H52yaaAdQOXq2YktTCZdlAcNKFzE6QtRz0snpw9bNiPeOIkkQvw==
dependencies:
node-fetch "2.6.7"
cross-spawn@^7.0.2, cross-spawn@^7.0.3:
version "7.0.3"
resolved "https://registry.yarnpkg.com/cross-spawn/-/cross-spawn-7.0.3.tgz#f73a85b9d5d41d045551c177e2882d4ac85728a6"
@ -1339,6 +1413,13 @@ crypto-browserify@^3.12.0:
randombytes "^2.0.0"
randomfill "^1.0.3"
debug@4, debug@4.3.4:
version "4.3.4"
resolved "https://registry.yarnpkg.com/debug/-/debug-4.3.4.tgz#1319f6579357f2338d3337d2cdd4914bb5dcc865"
integrity sha512-PRWFHuSU3eDtQJPvnNY7Jcket1j0t5OuOsFzPPzsekD52Zl8qUfFIPEiswXqIvHWGVHOgX+7G/vCNNhehwxfkQ==
dependencies:
ms "2.1.2"
debug@^4.0.1, debug@^4.1.0, debug@^4.1.1, debug@^4.3.1:
version "4.3.1"
resolved "https://registry.yarnpkg.com/debug/-/debug-4.3.1.tgz#f0d229c505e0c6d8c49ac553d1b13dc183f6b2ee"
@ -1403,6 +1484,11 @@ detect-newline@^3.0.0:
resolved "https://registry.yarnpkg.com/detect-newline/-/detect-newline-3.1.0.tgz#576f5dfc63ae1a192ff192d8ad3af6308991b651"
integrity sha512-TLz+x/vEXm/Y7P7wn1EJFNLxYpUD4TgMosxY6fAVJUnJMbupHBOncxyWUG9OpTaH9EBD7uFI5LfEgmMOc54DsA==
devtools-protocol@0.0.1107588:
version "0.0.1107588"
resolved "https://registry.yarnpkg.com/devtools-protocol/-/devtools-protocol-0.0.1107588.tgz#f8cac707840b97cc30b029359341bcbbb0ad8ffa"
integrity sha512-yIR+pG9x65Xko7bErCUSQaDLrO/P1p3JUzEk7JCU4DowPcGHkTGUGQapcfcLc4qj0UaALwZ+cr0riFgiqpixcg==
diff-sequences@^29.2.0:
version "29.2.0"
resolved "https://registry.yarnpkg.com/diff-sequences/-/diff-sequences-29.2.0.tgz#4c55b5b40706c7b5d2c5c75999a50c56d214e8f6"
@ -1459,7 +1545,7 @@ emoji-regex@^8.0.0:
resolved "https://registry.yarnpkg.com/emoji-regex/-/emoji-regex-8.0.0.tgz#e818fd69ce5ccfcb404594f842963bf53164cc37"
integrity sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==
end-of-stream@^1.1.0:
end-of-stream@^1.1.0, end-of-stream@^1.4.1:
version "1.4.4"
resolved "https://registry.yarnpkg.com/end-of-stream/-/end-of-stream-1.4.4.tgz#5ae64a5f45057baf3626ec14da0ca5e4b2431eb0"
integrity sha512-+uw1inIHVPQoaVuHzRyXd21icM+cnt4CzD5rW+NC1wjOUSTOs+Te7FOv7AhN7vS9x/oIyhLP5PR1H+phQAHu5Q==
@ -1725,6 +1811,17 @@ expect@^29.2.1:
jest-message-util "^29.2.1"
jest-util "^29.2.1"
extract-zip@2.0.1:
version "2.0.1"
resolved "https://registry.yarnpkg.com/extract-zip/-/extract-zip-2.0.1.tgz#663dca56fe46df890d5f131ef4a06d22bb8ba13a"
integrity sha512-GDhU9ntwuKyGXdZBUgTIe+vXnWj0fppUEtMDL0+idd5Sta8TGpHssn/eusA9mrPr9qNDym6SxAYZjNvCn/9RBg==
dependencies:
debug "^4.1.1"
get-stream "^5.1.0"
yauzl "^2.10.0"
optionalDependencies:
"@types/yauzl" "^2.9.1"
fast-deep-equal@^3.1.1:
version "3.1.3"
resolved "https://registry.yarnpkg.com/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz#3a7d56b559d6cbc3eb512325244e619a65c6c525"
@ -1752,6 +1849,13 @@ fb-watchman@^2.0.0:
dependencies:
bser "2.1.1"
fd-slicer@~1.1.0:
version "1.1.0"
resolved "https://registry.yarnpkg.com/fd-slicer/-/fd-slicer-1.1.0.tgz#25c7c89cb1f9077f8891bbe61d8f390eae256f1e"
integrity sha512-cE1qsB/VwyQozZ+q1dGxR8LBYNZeofhEdUNGSMbQD3Gw2lAzX9Zb3uIU6Ebc/Fmyjo9AWWfnn0AUCHqtevs/8g==
dependencies:
pend "~1.2.0"
file-entry-cache@^6.0.1:
version "6.0.1"
resolved "https://registry.yarnpkg.com/file-entry-cache/-/file-entry-cache-6.0.1.tgz#211b2dd9659cb0394b073e7323ac3c933d522027"
@ -1792,6 +1896,11 @@ foreach@^2.0.5:
resolved "https://registry.yarnpkg.com/foreach/-/foreach-2.0.5.tgz#0bee005018aeb260d0a3af3ae658dd0136ec1b99"
integrity sha1-C+4AUBiusmDQo6865ljdATbsG5k=
fs-constants@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/fs-constants/-/fs-constants-1.0.0.tgz#6be0de9be998ce16af8afc24497b9ee9b7ccd9ad"
integrity sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==
fs.realpath@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/fs.realpath/-/fs.realpath-1.0.0.tgz#1504ad2523158caa40db4a2787cb01411994ea4f"
@ -2011,11 +2120,24 @@ http2-wrapper@^1.0.0-beta.5.2:
quick-lru "^5.1.1"
resolve-alpn "^1.0.0"
https-proxy-agent@5.0.1:
version "5.0.1"
resolved "https://registry.yarnpkg.com/https-proxy-agent/-/https-proxy-agent-5.0.1.tgz#c59ef224a04fe8b754f3db0063a25ea30d0005d6"
integrity sha512-dFcAjpTQFgoLMzC2VwU+C/CbS7uRL0lWmxDITmqm7C+7F0Odmj6s9l6alZc6AELXhrnggM2CeWSXHGOdX2YtwA==
dependencies:
agent-base "6"
debug "4"
human-signals@^2.1.0:
version "2.1.0"
resolved "https://registry.yarnpkg.com/human-signals/-/human-signals-2.1.0.tgz#dc91fcba42e4d06e4abaed33b3e7a3c02f514ea0"
integrity sha512-B4FFZ6q/T2jhhksgkbEW3HBvWIfDW85snkQgawt07S7J5QXTk6BkNV+0yAeZrM5QpMAdYlocGoljn0sJ/WQkFw==
ieee754@^1.1.13:
version "1.2.1"
resolved "https://registry.yarnpkg.com/ieee754/-/ieee754-1.2.1.tgz#8eb7a10a63fff25d15a57b001586d177d1b0d352"
integrity sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==
ignore@^4.0.6:
version "4.0.6"
resolved "https://registry.yarnpkg.com/ignore/-/ignore-4.0.6.tgz#750e3db5862087b4737ebac8207ffd1ef27b25fc"
@ -2937,6 +3059,16 @@ minio@7.0.26:
xml "^1.0.0"
xml2js "^0.4.15"
mitt@3.0.0:
version "3.0.0"
resolved "https://registry.yarnpkg.com/mitt/-/mitt-3.0.0.tgz#69ef9bd5c80ff6f57473e8d89326d01c414be0bd"
integrity sha512-7dX2/10ITVyqh4aOSVI9gdape+t9l2/8QxHrFmUXu4EEUpdlxl6RudZUPZoc+zuY2hk1j7XxVroIVIan/pD/SQ==
mkdirp-classic@^0.5.2:
version "0.5.3"
resolved "https://registry.yarnpkg.com/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz#fa10c9115cc6d8865be221ba47ee9bed78601113"
integrity sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==
mkdirp@^0.5.1:
version "0.5.5"
resolved "https://registry.yarnpkg.com/mkdirp/-/mkdirp-0.5.5.tgz#d91cefd62d1436ca0f41620e251288d420099def"
@ -2954,6 +3086,13 @@ natural-compare@^1.4.0:
resolved "https://registry.yarnpkg.com/natural-compare/-/natural-compare-1.4.0.tgz#4abebfeed7541f2c27acfb29bdbbd15c8d5ba4f7"
integrity sha1-Sr6/7tdUHywnrPspvbvRXI1bpPc=
node-fetch@2.6.7:
version "2.6.7"
resolved "https://registry.yarnpkg.com/node-fetch/-/node-fetch-2.6.7.tgz#24de9fba827e3b4ae44dc8b20256a379160052ad"
integrity sha512-ZjMPFEfVx5j+y2yF35Kzx5sF7kDzxuDj6ziH4FFbOp87zKDZNx8yExJIb05OGF4Nlt9IHFIMBkRl41VdvcNdbQ==
dependencies:
whatwg-url "^5.0.0"
node-int64@^0.4.0:
version "0.4.0"
resolved "https://registry.yarnpkg.com/node-int64/-/node-int64-0.4.0.tgz#87a9065cdb355d3182d8f94ce11188b825c68a3b"
@ -3172,6 +3311,11 @@ pbkdf2@^3.0.3:
safe-buffer "^5.0.1"
sha.js "^2.4.8"
pend@~1.2.0:
version "1.2.0"
resolved "https://registry.yarnpkg.com/pend/-/pend-1.2.0.tgz#7a57eb550a6783f9115331fcf4663d5c8e007a50"
integrity sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg==
picocolors@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/picocolors/-/picocolors-1.0.0.tgz#cb5bdc74ff3f51892236eaf79d68bc44564ab81c"
@ -3199,11 +3343,6 @@ pkg-dir@^4.2.0:
dependencies:
find-up "^4.0.0"
playwright-core@^1.31.2:
version "1.31.2"
resolved "https://registry.yarnpkg.com/playwright-core/-/playwright-core-1.31.2.tgz#debf4b215d14cb619adb7e511c164d068075b2ed"
integrity sha512-a1dFgCNQw4vCsG7bnojZjDnPewZcw7tZUNFN0ZkcLYKj+mPmXvg4MpaaKZ5SgqPsOmqIf2YsVRkgqiRDxD+fDQ==
prelude-ls@^1.2.1:
version "1.2.1"
resolved "https://registry.yarnpkg.com/prelude-ls/-/prelude-ls-1.2.1.tgz#debc6489d7a6e6b0e7611888cec880337d316396"
@ -3218,7 +3357,7 @@ pretty-format@^29.2.1:
ansi-styles "^5.0.0"
react-is "^18.0.0"
progress@^2.0.0:
progress@2.0.3, progress@^2.0.0:
version "2.0.3"
resolved "https://registry.yarnpkg.com/progress/-/progress-2.0.3.tgz#7e8cf8d8f5b8f239c1bc68beb4eb78567d572ef8"
integrity sha512-7PiHtLll5LdnKIMw100I+8xJXR5gW2QwWYkT6iJva0bXitZKa/XMrSbdmg3r2Xnaidz9Qumd0VPaMrZlF9V9sA==
@ -3240,6 +3379,11 @@ prop-types@^15.7.2:
object-assign "^4.1.1"
react-is "^16.8.1"
proxy-from-env@1.1.0:
version "1.1.0"
resolved "https://registry.yarnpkg.com/proxy-from-env/-/proxy-from-env-1.1.0.tgz#e102f16ca355424865755d2c9e8ea4f24d58c3e2"
integrity sha512-D+zkORCbA9f1tdWRK0RaCR3GPv50cMxcrz4X8k5LTSUD1Dkw47mKJEZQNunItRTkWwgtaUSo1RVFRIG9ZXiFYg==
public-encrypt@^4.0.0:
version "4.0.3"
resolved "https://registry.yarnpkg.com/public-encrypt/-/public-encrypt-4.0.3.tgz#4fcc9d77a07e48ba7527e7cbe0de33d0701331e0"
@ -3265,6 +3409,23 @@ punycode@^2.1.0:
resolved "https://registry.yarnpkg.com/punycode/-/punycode-2.1.1.tgz#b58b010ac40c22c5657616c8d2c2c02c7bf479ec"
integrity sha512-XRsRjdf+j5ml+y/6GKHPZbrF/8p2Yga0JPtdqTIY2Xe5ohJPD9saDJJLPvp9+NSBprVvevdXZybnj2cv8OEd0A==
puppeteer-core@^19.11.1:
version "19.11.1"
resolved "https://registry.yarnpkg.com/puppeteer-core/-/puppeteer-core-19.11.1.tgz#4c63d7a0a6cd268ff054ebcac315b646eee32667"
integrity sha512-qcuC2Uf0Fwdj9wNtaTZ2OvYRraXpAK+puwwVW8ofOhOgLPZyz1c68tsorfIZyCUOpyBisjr+xByu7BMbEYMepA==
dependencies:
"@puppeteer/browsers" "0.5.0"
chromium-bidi "0.4.7"
cross-fetch "3.1.5"
debug "4.3.4"
devtools-protocol "0.0.1107588"
extract-zip "2.0.1"
https-proxy-agent "5.0.1"
proxy-from-env "1.1.0"
tar-fs "2.1.1"
unbzip2-stream "1.4.3"
ws "8.13.0"
querystring@0.2.0:
version "0.2.0"
resolved "https://registry.yarnpkg.com/querystring/-/querystring-0.2.0.tgz#b209849203bb25df820da756e747005878521620"
@ -3309,6 +3470,15 @@ react-is@^18.0.0:
string_decoder "^1.1.1"
util-deprecate "^1.0.1"
readable-stream@^3.1.1:
version "3.6.2"
resolved "https://registry.yarnpkg.com/readable-stream/-/readable-stream-3.6.2.tgz#56a9b36ea965c00c5a93ef31eb111a0f11056967"
integrity sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==
dependencies:
inherits "^2.0.3"
string_decoder "^1.1.1"
util-deprecate "^1.0.1"
redis-commands@1.7.0:
version "1.7.0"
resolved "https://registry.yarnpkg.com/redis-commands/-/redis-commands-1.7.0.tgz#15a6fea2d58281e27b1cd1acfb4b293e278c3a89"
@ -3684,6 +3854,27 @@ table@^6.0.4:
string-width "^4.2.0"
strip-ansi "^6.0.0"
tar-fs@2.1.1:
version "2.1.1"
resolved "https://registry.yarnpkg.com/tar-fs/-/tar-fs-2.1.1.tgz#489a15ab85f1f0befabb370b7de4f9eb5cbe8784"
integrity sha512-V0r2Y9scmbDRLCNex/+hYzvp/zyYjvFbHPNgVTKfQvVrb6guiE/fxP+XblDNR011utopbkex2nM4dHNV6GDsng==
dependencies:
chownr "^1.1.1"
mkdirp-classic "^0.5.2"
pump "^3.0.0"
tar-stream "^2.1.4"
tar-stream@^2.1.4:
version "2.2.0"
resolved "https://registry.yarnpkg.com/tar-stream/-/tar-stream-2.2.0.tgz#acad84c284136b060dc3faa64474aa9aebd77287"
integrity sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==
dependencies:
bl "^4.0.3"
end-of-stream "^1.4.1"
fs-constants "^1.0.0"
inherits "^2.0.3"
readable-stream "^3.1.1"
test-exclude@^6.0.0:
version "6.0.0"
resolved "https://registry.yarnpkg.com/test-exclude/-/test-exclude-6.0.0.tgz#04a8698661d805ea6fa293b6cb9e63ac044ef15e"
@ -3706,6 +3897,11 @@ through2@^3.0.1:
inherits "^2.0.4"
readable-stream "2 || 3"
through@^2.3.8:
version "2.3.8"
resolved "https://registry.yarnpkg.com/through/-/through-2.3.8.tgz#0dd4c9ffaabc357960b1b724115d7e0e86a2e1f5"
integrity sha512-w89qg7PI8wAdvX60bMDP+bFoD5Dvhm9oLheFp5O4a2QF0cSBGsBX4qZmadPMvVqlLJBBci+WqGGOAPvcDeNSVg==
tmpl@1.0.5:
version "1.0.5"
resolved "https://registry.yarnpkg.com/tmpl/-/tmpl-1.0.5.tgz#8683e0b902bb9c20c4f726e3c0b69f36518c07cc"
@ -3723,6 +3919,11 @@ to-regex-range@^5.0.1:
dependencies:
is-number "^7.0.0"
tr46@~0.0.3:
version "0.0.3"
resolved "https://registry.yarnpkg.com/tr46/-/tr46-0.0.3.tgz#8184fd347dac9cdc185992f3a6622e14b9d9ab6a"
integrity sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw==
type-check@^0.4.0, type-check@~0.4.0:
version "0.4.0"
resolved "https://registry.yarnpkg.com/type-check/-/type-check-0.4.0.tgz#07b8203bfa7056c0657050e3ccd2c37730bab8f1"
@ -3760,6 +3961,14 @@ unbox-primitive@^1.0.0, unbox-primitive@^1.0.1:
has-symbols "^1.0.2"
which-boxed-primitive "^1.0.2"
unbzip2-stream@1.4.3:
version "1.4.3"
resolved "https://registry.yarnpkg.com/unbzip2-stream/-/unbzip2-stream-1.4.3.tgz#b0da04c4371311df771cdc215e87f2130991ace7"
integrity sha512-mlExGW4w71ebDJviH16lQLtZS32VKqsSfk80GCfUlwT/4/hNRFsoscrF/c++9xinkMzECL1uL9DDwXqFWkruPg==
dependencies:
buffer "^5.2.1"
through "^2.3.8"
update-browserslist-db@^1.0.9:
version "1.0.10"
resolved "https://registry.yarnpkg.com/update-browserslist-db/-/update-browserslist-db-1.0.10.tgz#0f54b876545726f17d00cd9a2561e6dade943ff3"
@ -3842,6 +4051,19 @@ web-encoding@^1.1.5:
optionalDependencies:
"@zxing/text-encoding" "0.9.0"
webidl-conversions@^3.0.0:
version "3.0.1"
resolved "https://registry.yarnpkg.com/webidl-conversions/-/webidl-conversions-3.0.1.tgz#24534275e2a7bc6be7bc86611cc16ae0a5654871"
integrity sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ==
whatwg-url@^5.0.0:
version "5.0.0"
resolved "https://registry.yarnpkg.com/whatwg-url/-/whatwg-url-5.0.0.tgz#966454e8765462e37644d3626f6742ce8b70965d"
integrity sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw==
dependencies:
tr46 "~0.0.3"
webidl-conversions "^3.0.0"
which-boxed-primitive@^1.0.2:
version "1.0.2"
resolved "https://registry.yarnpkg.com/which-boxed-primitive/-/which-boxed-primitive-1.0.2.tgz#13757bc89b209b049fe5d86430e21cf40a89a8e6"
@ -3913,6 +4135,11 @@ write-file-atomic@^4.0.1:
imurmurhash "^0.1.4"
signal-exit "^3.0.7"
ws@8.13.0:
version "8.13.0"
resolved "https://registry.yarnpkg.com/ws/-/ws-8.13.0.tgz#9a9fb92f93cf41512a0735c8f4dd09b8a1211cd0"
integrity sha512-x9vcZYTrFPC7aSIbj7sRCYo7L/Xb8Iy+pW0ng0wt2vCJv7M9HOMy0UoN3rr+IFC7hb7vXoqS+P9ktyLLLhO+LA==
ws@^7.4.4:
version "7.5.9"
resolved "https://registry.yarnpkg.com/ws/-/ws-7.5.9.tgz#54fa7db29f4c7cec68b1ddd3a89de099942bb591"
@ -3964,11 +4191,24 @@ yargs-parser@^20.2.2:
resolved "https://registry.yarnpkg.com/yargs-parser/-/yargs-parser-20.2.9.tgz#2eb7dc3b0289718fc295f362753845c41a0c94ee"
integrity sha512-y11nGElTIV+CT3Zv9t7VKl+Q3hTQoT9a1Qzezhhl6Rp21gJ/IVTW7Z3y9EWXhuUBC2Shnf+DX0antecpAwSP8w==
yargs-parser@^21.0.0:
yargs-parser@^21.0.0, yargs-parser@^21.1.1:
version "21.1.1"
resolved "https://registry.yarnpkg.com/yargs-parser/-/yargs-parser-21.1.1.tgz#9096bceebf990d21bb31fa9516e0ede294a77d35"
integrity sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==
yargs@17.7.1:
version "17.7.1"
resolved "https://registry.yarnpkg.com/yargs/-/yargs-17.7.1.tgz#34a77645201d1a8fc5213ace787c220eabbd0967"
integrity sha512-cwiTb08Xuv5fqF4AovYacTFNxk62th7LKJ6BL9IGUpTJrWoU7/7WdQGTP2SjKf1dUNBGzDd28p/Yfs/GI6JrLw==
dependencies:
cliui "^8.0.1"
escalade "^3.1.1"
get-caller-file "^2.0.5"
require-directory "^2.1.1"
string-width "^4.2.3"
y18n "^5.0.5"
yargs-parser "^21.1.1"
yargs@^15.3.1:
version "15.4.1"
resolved "https://registry.yarnpkg.com/yargs/-/yargs-15.4.1.tgz#0d87a16de01aee9d8bec2bfbf74f67851730f4f8"
@ -4012,6 +4252,14 @@ yargs@^17.3.1:
y18n "^5.0.5"
yargs-parser "^21.0.0"
yauzl@^2.10.0:
version "2.10.0"
resolved "https://registry.yarnpkg.com/yauzl/-/yauzl-2.10.0.tgz#c7eb17c93e112cb1086fa6d8e51fb0667b79a5f9"
integrity sha512-p4a9I6X6nu6IhoGmBqAcbJy1mlC4j27vEPZX9F4L4/vZT3Lyq1VkFHw/V/PUcB9Buo+DG3iHkT0x3Qya58zc3g==
dependencies:
buffer-crc32 "~0.2.3"
fd-slicer "~1.1.0"
yocto-queue@^0.1.0:
version "0.1.0"
resolved "https://registry.yarnpkg.com/yocto-queue/-/yocto-queue-0.1.0.tgz#0294eb3dee05028d31ee1a5fa2c556a6aaf10a1b"