Switch back to Puppeteer from Playwright (#301)

- reduced memory usage, avoids memory leak issues caused by using playwright (see #298) 
- browser: split Browser into Browser and BaseBrowser
- browser: puppeteer-specific functions added to Browser for additional flexibility if need to change again later
- browser: use defaultArgs from playwright
- browser: attempt to recover if initial target is gone
- logging: add debug logging from process.memoryUsage() after every page
- request interception: use priorities for cooperative request interception
- request interception: move to setupPage() to run once per page, enable if any of blockrules, adblockrules or originOverrides are used
- request interception: fix originOverrides enabled check, fix to work with catch-all request interception
- default args: set --waitUntil back to 'load,networkidle2'
- Update README with changes for puppeteer
- tests: fix extra hops depth test to ensure more than one page crawled

---------
Co-authored-by: Tessa Walsh <tessa@bitarchivist.net>

README.md

@@ -1,6 +1,6 @@
 # Browsertrix Crawler
 
-Browsertrix Crawler is a simplified (Chrome) browser-based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker container. Browsertrix Crawler uses [Playwright](https://github.com/microsoft/playwright) to control one or more browser windows in parallel.
+Browsertrix Crawler is a simplified (Chrome) browser-based high-fidelity crawling system, designed to run a complex, customizable browser-based crawl in a single Docker container. Browsertrix Crawler uses [Puppeteer](https://github.com/puppeteer/puppeteer) to control one or more browser windows in parallel.
 
 ## Features
 
@@ -14,7 +14,7 @@ Thus far, Browsertrix Crawler supports:
 - Screencasting: Ability to watch crawling in real-time (experimental).
 - Screenshotting: Ability to take thumbnails, full page screenshots, and/or screenshots of the initial page view.
 - Optimized (non-browser) capture of non-HTML resources.
-- Extensible Playwright driver script for customizing behavior per crawl or page.
+- Extensible Puppeteer driver script for customizing behavior per crawl or page.
 - Ability to create and reuse browser profiles interactively or via automated user/password login using an embedded browser.
 - Multi-platform support -- prebuilt Docker images available for Intel/AMD and Apple Silicon (M1/M2) CPUs.
 
@@ -69,13 +69,14 @@ Options:
       --crawlId, --id                       A user provided ID for this crawl or
                                              crawl configuration (can also be se
                                             t via CRAWL_ID env var)
-                                              [string] [default: "454230b33b8f"]
+                                              [string] [default: "97792ef37eaf"]
       --newContext                          Deprecated as of 0.8.0, any values p
                                             assed will be ignored
                                                         [string] [default: null]
-      --waitUntil                           Playwright page.goto() condition to
+      --waitUntil                           Puppeteer page.goto() condition to w
-                                            wait for before continuing
+                                            ait for before continuing, can be mu
-                                                               [default: "load"]
+                                            ltiple separated by ','
+                                                  [default: "load,networkidle2"]
       --depth                               The depth of the crawl for all seeds
                                                           [number] [default: -1]
       --extraHops                           Number of extra 'hops' to follow, be
@@ -150,10 +151,9 @@ Options:
                                             o process.cwd()
                                                    [string] [default: "/crawls"]
       --mobileDevice                        Emulate mobile device by name from:
-                                            https://github.com/microsoft/playwri
+                                            https://github.com/puppeteer/puppete
-                                            ght/blob/main/packages/playwright-co
+                                            er/blob/main/src/common/DeviceDescri
-                                            re/src/server/deviceDescriptorsSourc
+                                            ptors.ts                    [string]
-                                            e.json                      [string]
       --userAgent                           Override user-agent with specified s
                                             tring                       [string]
       --userAgentSuffix                     Append suffix to existing browser us
@@ -240,6 +240,13 @@ Options:
       --description, --desc                 If set, write supplied description i
                                             nto WACZ datapackage.json metadata
                                                                         [string]
+      --originOverride                      if set, will redirect requests from
+                                            each origin in key to origin in the
+                                            value, eg. --originOverride https://
+                                            host:port=http://alt-host:alt-port
+                                                           [array] [default: []]
+      --logErrorsToRedis                    If set, write error messages to redi
+                                            s         [boolean] [default: false]
       --config                              Path to YAML config file
 
 ```
@@ -250,9 +257,9 @@ Options:
 
 One of the key nuances of browser-based crawling is determining when a page is finished loading. This can be configured with the `--waitUntil` flag.
 
-The default is `load`, which waits until page load, but for static sites, `--wait-until domcontentloaded` may be used to speed up the crawl (to avoid waiting for ads to load for example). The `--waitUntil networkidle` may make sense for sites where absolutely all requests must be waited until before proceeding.
+The default is `load,networkidle2`, which waits until page load and <=2 requests remain, but for static sites, `--wait-until domcontentloaded` may be used to speed up the crawl (to avoid waiting for ads to load for example). `--waitUntil networkidle0` may make sense for sites where absolutely all requests must be waited until before proceeding.
 
-See [page.goto waitUntil options](https://playwright.dev/docs/api/class-page#page-goto-option-wait-until) for more info on the options that can be used with this flag from the Playwright docs.
+See [page.goto waitUntil options](https://pptr.dev/api/puppeteer.page.goto#remarks) for more info on the options that can be used with this flag from the Puppeteer docs.
 
 The `--pageLoadTimeout`/`--timeout` option sets the timeout in seconds for page load, defaulting to 90 seconds. Behaviors will run on the page once either the page load condition or the page load timeout is met, whichever happens first.
 
@@ -543,11 +550,11 @@ The webhook URL can be an HTTP URL which receives a JSON POST request OR a Redis
 
 </details>
 
-### Configuring Chromium / Playwright / pywb
+### Configuring Chromium / Puppeteer / pywb
 
 There is a few environment variables you can set to configure chromium and pywb:
 
-- CHROME_FLAGS will be split by spaces and passed to Chromium (via `args` in Playwright). Note that setting some options is not supported such as `--proxy-server` since they are set by browsertrix itself.
+- CHROME_FLAGS will be split by spaces and passed to Chromium (via `args` in Puppeteer). Note that setting some options is not supported such as `--proxy-server` since they are set by browsertrix itself.
 - SOCKS_HOST and SOCKS_PORT are read by pywb to proxy upstream traffic
 
 Here's some examples use cases:

crawler.js

@@ -355,6 +355,24 @@ export class Crawler {
   async setupPage({page, cdp, workerid}) {
     await this.browser.setupPage({page, cdp});
 
+    if ((this.adBlockRules && this.params.blockAds) ||
+        this.blockRules || this.originOverride) {
+
+      await page.setRequestInterception(true);
+
+      if (this.adBlockRules && this.params.blockAds) {
+        await this.adBlockRules.initPage(this.browser, page);
+      }
+
+      if (this.blockRules) {
+        await this.blockRules.initPage(this.browser, page);
+      }
+
+      if (this.originOverride) {
+        await this.originOverride.initPage(this.browser, page);
+      }
+    }
+
     if (this.params.logging.includes("jserrors")) {
       page.on("console", (msg) => {
         if (msg.type() === "error") {
@@ -374,7 +392,7 @@ export class Crawler {
 
     if (this.params.behaviorOpts) {
       await page.exposeFunction(BEHAVIOR_LOG_FUNC, (logdata) => this._behaviorLog(logdata, page.url(), workerid));
-      await page.addInitScript(behaviors + `;\nself.__bx_behaviors.init(${this.params.behaviorOpts});`);
+      await this.browser.addInitScript(page, behaviors + `;\nself.__bx_behaviors.init(${this.params.behaviorOpts});`);
     }
   }
 
@@ -404,7 +422,7 @@ export class Crawler {
         logger.debug("Skipping screenshots for non-HTML page", logDetails);
       }
       const archiveDir = path.join(this.collDir, "archive");
-      const screenshots = new Screenshots({page, url, directory: archiveDir});
+      const screenshots = new Screenshots({browser: this.browser, page, url, directory: archiveDir});
       if (this.params.screenshot.includes("view")) {
         await screenshots.take();
       }
@@ -430,7 +448,7 @@ export class Crawler {
         logger.info("Skipping behaviors for slow page", logDetails, "behavior");
       } else {
         const res = await timedRun(
-          this.runBehaviors(page, data.filteredFrames, logDetails),
+          this.runBehaviors(page, cdp, data.filteredFrames, logDetails),
           this.params.behaviorTimeout,
           "Behaviors timed out",
           logDetails,
@@ -495,16 +513,14 @@ export class Crawler {
     }
   }
 
-  async runBehaviors(page, frames, logDetails) {
+  async runBehaviors(page, cdp, frames, logDetails) {
     try {
       frames = frames || page.frames();
 
-      const context = page.context();
-
       logger.info("Running behaviors", {frames: frames.length, frameUrls: frames.map(frame => frame.url()), ...logDetails}, "behavior");
 
       return await Promise.allSettled(
-        frames.map(frame => this.browser.evaluateWithCLI(context, frame, "self.__bx_behaviors.run();", logDetails, "behavior"))
+        frames.map(frame => this.browser.evaluateWithCLI(page, frame, cdp, "self.__bx_behaviors.run();", logDetails, "behavior"))
       );
 
     } catch (e) {
@@ -711,7 +727,7 @@ export class Crawler {
 
     this.screencaster = this.initScreenCaster();
 
-    if (this.params.originOverride) {
+    if (this.params.originOverride.length) {
       this.originOverride = new OriginOverride(this.params.originOverride);
     }
 
@@ -905,6 +921,14 @@ export class Crawler {
     });
   }
 
+  logMemory() {
+    const memUsage = process.memoryUsage();
+    const { heapUsed, heapTotal } = memUsage;
+    this.maxHeapUsed = Math.max(this.maxHeapUsed || 0, heapUsed);
+    this.maxHeapTotal = Math.max(this.maxHeapTotal || 0, heapTotal);
+    logger.debug("Memory", {maxHeapUsed: this.maxHeapUsed, maxHeapTotal: this.maxHeapTotal, ...memUsage}, "memory");
+  }
+
   async writeStats(toFile=false) {
     if (!this.params.logging.includes("stats")) {
       return;
@@ -926,6 +950,7 @@ export class Crawler {
     };
 
     logger.info("Crawl statistics", stats, "crawlStatus");
+    this.logMemory();
 
     if (toFile && this.params.statsFilename) {
       try {
@@ -965,18 +990,6 @@ export class Crawler {
       }
     }
 
-    if (this.adBlockRules && this.params.blockAds) {
-      await this.adBlockRules.initPage(page);
-    }
-
-    if (this.blockRules) {
-      await this.blockRules.initPage(page);
-    }
-
-    if (this.originOverride) {
-      await this.originOverride.initPage(page);
-    }
-
     let ignoreAbort = false;
 
     // Detect if ERR_ABORTED is actually caused by trying to load a non-page (eg. downloadable PDF),
@@ -998,7 +1011,7 @@ export class Crawler {
     try {
       const resp = await page.goto(url, gotoOpts);
 
-      const contentType = await resp.headerValue("content-type");
+      const contentType = await this.browser.responseHeader(resp, "content-type");
 
       isHTMLPage = this.isHTMLContentType(contentType);
 
@@ -1068,7 +1081,7 @@ export class Crawler {
     await sleep(0.5);
 
     try {
-      await page.waitForLoadState("networkidle", {timeout: this.params.netIdleWait * 1000});
+      await this.browser.waitForNetworkIdle(page, {timeout: this.params.netIdleWait * 1000});
     } catch (e) {
       logger.debug("waitForNetworkIdle timed out, ignoring", details);
       // ignore, continue
@@ -1095,7 +1108,7 @@ export class Crawler {
     try {
       const linkResults = await Promise.allSettled(
         frames.map(frame => timedRun(
-          frame.evaluate(loadFunc, {selector: selector, extract: extract}),
+          frame.evaluate(loadFunc, {selector, extract}),
           PAGE_OP_TIMEOUT_SECS,
           "Link extraction timed out",
           logDetails,
@@ -1152,9 +1165,10 @@ export class Crawler {
     try {
       logger.debug("Check CF Blocking", logDetails);
 
-      const cloudflare = page.locator("div.cf-browser-verification.cf-im-under-attack");
+      while (await timedRun(
-
+        page.$("div.cf-browser-verification.cf-im-under-attack"),
-      while (await cloudflare.waitFor({timeout: PAGE_OP_TIMEOUT_SECS})) {
+        PAGE_OP_TIMEOUT_SECS
+      )) {
         logger.debug("Cloudflare Check Detected, waiting for reload...", logDetails);
         await sleep(5.5);
       }

create-login-profile.js

@@ -158,10 +158,8 @@ async function main() {
 
   const browser = new Browser();
 
-  const profileDir = await browser.loadProfile(params.profile);
-
   await browser.launch({
-    dataDir: profileDir,
+    profileUrl: params.profile,
     headless: params.headless,
     signals: true,
     chromeOptions: {
@@ -191,18 +189,17 @@ async function main() {
     params.password = await promptInput("Enter password: ", true);
   }
 
-  const { page, cdp } = await browser.getFirstPageWithCDP();
+  const { page, cdp } = await browser.newWindowPageWithCDP();
 
   const waitUntil = "load";
 
-  //await page.setCacheEnabled(false);
+  await page.setCacheEnabled(false);
-  await cdp.send("Network.setCacheDisabled", {cacheDisabled: true});
 
   if (!params.automated) {
     await browser.setupPage({page, cdp});
 
     // for testing, inject browsertrix-behaviors
-    await page.addInitScript(behaviors + ";\nself.__bx_behaviors.init();");
+    await browser.addInitScript(page, behaviors + ";\nself.__bx_behaviors.init();");
   }
 
   logger.info(`Loading page: ${params.url}`);
@@ -384,7 +381,7 @@ class InteractiveBrowser {
         return;
       }
 
-      const cookies = await this.browser.context.cookies(url);
+      const cookies = await this.browser.getCookies(this.page, url);
       for (const cookie of cookies) {
         cookie.expires = (new Date().getTime() / 1000) + this.params.cookieDays * 86400;
         delete cookie.size;
@@ -396,7 +393,7 @@ class InteractiveBrowser {
           cookie.url = url;
         }
       }
-      await this.browser.context.addCookies(cookies);
+      await this.browser.setCookies(this.page, cookies);
     } catch (e) {
       logger.error("Save Cookie Error: ", e);
     }

package.json

@@ -17,7 +17,7 @@
     "ioredis": "^4.27.1",
     "js-yaml": "^4.1.0",
     "minio": "7.0.26",
-    "playwright-core": "^1.31.2",
+    "puppeteer-core": "^19.11.1",
     "sitemapper": "^3.1.2",
     "uuid": "8.3.2",
     "warcio": "^1.6.0",

tests/extra_hops_depth.test.js

@@ -16,7 +16,8 @@ test("check that URLs are crawled 2 extra hops beyond depth", async () => {
     console.log(error);
   }
 
-  const crawled_pages = fs.readFileSync("test-crawls/collections/extra-hops-beyond/pages/pages.jsonl", "utf8");
+  const crawledPages = fs.readFileSync("test-crawls/collections/extra-hops-beyond/pages/pages.jsonl", "utf8");
+  const crawledPagesArray = crawledPages.trim().split("\n");
 
   const expectedPages = [
     "https://webrecorder.net/",
@@ -28,7 +29,10 @@ test("check that URLs are crawled 2 extra hops beyond depth", async () => {
     "https://webrecorder.net/faq",
   ];
 
-  for (const page of crawled_pages.trim().split("\n")) {
+  // first line is the header, not page, so adding -1
+  expect(expectedPages.length).toEqual(crawledPagesArray.length - 1);
+
+  for (const page of crawledPagesArray) {
     const url = JSON.parse(page).url;
     if (!url) {
       continue;

util/argParser.js

@@ -3,7 +3,7 @@ import fs from "fs";
 import os from "os";
 
 import yaml from "js-yaml";
-import { devices } from "playwright-core";
+import { KnownDevices as devices } from "puppeteer-core";
 import yargs from "yargs";
 import { hideBin } from "yargs/helpers";
 
@@ -52,8 +52,8 @@ class ArgParser {
       },
 
       "waitUntil": {
-        describe: "Playwright page.goto() condition to wait for before continuing",
+        describe: "Puppeteer page.goto() condition to wait for before continuing, can be multiple separated by ','",
-        default: "load",
+        default: "load,networkidle2",
       },
 
       "depth": {
@@ -208,7 +208,7 @@ class ArgParser {
       },
 
       "mobileDevice": {
-        describe: "Emulate mobile device by name from: https://github.com/microsoft/playwright/blob/main/packages/playwright-core/src/server/deviceDescriptorsSource.json",
+        describe: "Emulate mobile device by name from: https://github.com/puppeteer/puppeteer/blob/main/src/common/DeviceDescriptors.ts",
         type: "string",
       },
 
@@ -369,10 +369,10 @@ class ArgParser {
       },
 
       "logErrorsToRedis": {
-        descripe: "If set, write error messages to redis",
+        describe: "If set, write error messages to redis",
         type: "boolean",
         default: false,
-      }
+      },
     };
   }
 
@@ -410,11 +410,18 @@ class ArgParser {
       logger.fatal(`\n${argv.collection} is an invalid collection name. Please supply a collection name only using alphanumeric characters and the following characters [_ - ]\n`);
     }
 
-    // waitUntil condition must be one of WAIT_UNTIL_OPTS: load, domcontentloaded, networkidle
+    // waitUntil condition must be: load, domcontentloaded, networkidle0, networkidle2
-    // (see: https://playwright.dev/docs/api/class-page#page-goto-option-wait-until)
+    // can be multiple separate by comma
-    if (!WAIT_UNTIL_OPTS.includes(argv.waitUntil)) {
+    // (see: https://github.com/puppeteer/puppeteer/blob/main/docs/api.md#pagegotourl-options)
+    if (typeof argv.waitUntil != "object"){
+      argv.waitUntil = argv.waitUntil.split(",");
+    }
+
+    for (const opt of argv.waitUntil) {
+      if (!WAIT_UNTIL_OPTS.includes(opt)) {
         logger.fatal("Invalid waitUntil option, must be one of: " + WAIT_UNTIL_OPTS.join(","));
       }
+    }
 
     // validate screenshot options
     if (argv.screenshot) {
@@ -453,9 +460,6 @@ class ArgParser {
       if (!argv.emulateDevice) {
         logger.fatal("Unknown device: " + argv.mobileDevice);
       }
-      if (argv.emulateDevice.defaultBrowserType !== "chromium") {
-        logger.fatal(`Device Browser: ${argv.emulateDevice.defaultBrowserType}\r\nSorry, only Chromium-based devices are supported at this time`);
-      }
     } else {
       argv.emulateDevice = {viewport: null};
     }

util/blockrules.js

@@ -68,29 +68,19 @@ export class BlockRules
     }
   }
 
-  async initPage(page) {
+  async initPage(browser, page) {
-    if (!this.rules.length) {
+    const onRequest = async (request) => {
-      return;
-    }
-
-    if (page._btrix_interceptionAdded) {
-      return true;
-    }
-
-    page._btrix_interceptionAdded = true;
-
-    await page.route("**/*", (route) => {
       const logDetails = {page: page.url()};
       try {
-        this.handleRequest(route, logDetails);
+        await this.handleRequest(request, logDetails);
       } catch (e) {
         logger.warn("Error handling request", {...errJSON(e), ...logDetails}, "blocking");
       }
-    });
+    };
+    await browser.interceptRequest(page, onRequest);
   }
 
-  async handleRequest(route, logDetails) {
+  async handleRequest(request, logDetails) {
-    const request = route.request();
     const url = request.url();
 
     let blockState;
@@ -99,9 +89,9 @@ export class BlockRules
       blockState = await this.shouldBlock(request, url, logDetails);
 
       if (blockState === BlockState.ALLOW) {
-        await route.continue();
+        await request.continue({}, 1);
       } else {
-        await route.abort("blockedbyclient");
+        await request.abort("blockedbyclient", 1);
       }
 
     } catch (e) {
@@ -236,23 +226,6 @@ export class AdBlockRules extends BlockRules
     this.adhosts = JSON.parse(fs.readFileSync(new URL(adhostsFilePath, import.meta.url)));
   }
 
-  async initPage(page) {
-    if (page._btrix_adInterceptionAdded) {
-      return true;
-    }
-
-    page._btrix_adInterceptionAdded = true;
-
-    await page.route("**/*", (route) => {
-      const logDetails = {page: page.url()};
-      try {
-        this.handleRequest(route, logDetails);
-      } catch (e) {
-        logger.warn("Error handling request", {...errJSON(e), ...logDetails}, "blocking");
-      }
-    });
-  }
-
   isAdUrl(url) {
     const fragments = url.split("/");
     const domain = fragments.length > 2 ? fragments[2] : null;

util/browser.js

@@ -9,36 +9,32 @@ import path from "path";
 import { logger } from "./logger.js";
 import { initStorage } from "./storage.js";
 
-import { chromium } from "playwright-core";
+import puppeteer from "puppeteer-core";
 
 
 // ==================================================================
-export class Browser
+export class BaseBrowser
 {
   constructor() {
-    this.context = null;
-
-    this.firstPage = null;
-    this.firstCDP = null;
-
     this.profileDir = fs.mkdtempSync(path.join(os.tmpdir(), "profile-"));
     this.customProfile = false;
+    this.emulateDevice = null;
   }
 
-  async launch({profileUrl, chromeOptions, signals = false, headless = false, emulateDevice = {viewport: null}} = {}) {
+  async launch({profileUrl, chromeOptions, signals = false, headless = false, emulateDevice = {}} = {}) {
-    if (this.context) {
+    if (this.isLaunched()) {
-      logger.warn("Context already inited", {}, "context");
+      return;
-      return this.context;
     }
 
     if (profileUrl) {
       this.customProfile = await this.loadProfile(profileUrl);
     }
 
+    this.emulateDevice = emulateDevice;
+
     const args = this.chromeArgs(chromeOptions);
 
     const launchOpts = {
-      ...emulateDevice,
       args,
       headless,
       executablePath: this.getBrowserExe(),
@@ -47,30 +43,17 @@ export class Browser
       handleSIGHUP: signals,
       handleSIGINT: signals,
       handleSIGTERM: signals,
-      serviceWorkers: "allow"
+
+      defaultViewport: null,
+      waitForInitialPage: false,
+      userDataDir: this.profileDir
     };
 
-    this.context = await chromium.launchPersistentContext(this.profileDir, launchOpts);
+    await this._init(launchOpts);
-
-    if (this.context.pages()) {
-      this.firstPage = this.context.pages()[0];
-    } else {
-      this.firstPage = await this.context.newPage();
-    }
-    this.firstCDP = await this.context.newCDPSession(this.firstPage);
-
-    return this.context;
-  }
-
-  async close() {
-    if (this.context) {
-      await this.context.close();
-      this.context = null;
-    }
   }
 
   async setupPage({page, cdp}) {
-    await page.addInitScript("Object.defineProperty(navigator, \"webdriver\", {value: false});");
+    await this.addInitScript(page, "Object.defineProperty(navigator, \"webdriver\", {value: false});");
 
     if (this.customProfile) {
       logger.info("Disabling Service Workers for profile", {}, "browser");
@@ -79,38 +62,6 @@ export class Browser
     }
   }
 
-  async getFirstPageWithCDP() {
-    return {page: this.firstPage, cdp: this.firstCDP};
-  }
-
-  numPages() {
-    return this.context ? this.context.pages().length : 0;
-  }
-
-  async newWindowPageWithCDP() {
-    // unique url to detect new pages
-    const startPage = "about:blank?_browsertrix" + Math.random().toString(36).slice(2);
-
-    const p = new Promise((resolve) => {
-      const listener = (page) => {
-        if (page.url() === startPage) {
-          resolve(page);
-          this.context.removeListener("page", listener);
-        }
-      };
-
-      this.context.on("page", listener);
-    });
-
-    await this.firstCDP.send("Target.createTarget", {url: startPage, newWindow: true});
-
-    const page = await p;
-
-    const cdp = await this.context.newCDPSession(page);
-
-    return {page, cdp};
-  }
-
   async loadProfile(profileFilename) {
     const targetFilename = "/tmp/profile.tar.gz";
 
@@ -157,6 +108,7 @@ export class Browser
   chromeArgs({proxy=true, userAgent=null, extraArgs=[]} = {}) {
     // Chrome Flags, including proxy server
     const args = [
+      ...defaultArgs,
       ...(process.env.CHROME_FLAGS ?? "").split(" ").filter(Boolean),
       //"--no-xshm", // needed for Chrome >80 (check if puppeteer adds automatically)
       "--no-sandbox",
@@ -200,21 +152,19 @@ export class Browser
     return null;
   }
 
-  async evaluateWithCLI(context, frame, funcString, logData, contextName) {
+  async evaluateWithCLI_(cdp, frame, cdpContextId, funcString, logData, contextName) {
     let details = {frameUrl: frame.url(), ...logData};
 
     logger.info("Run Script Started", details, contextName);
 
-    const cdp = await context.newCDPSession(frame);
-
     // from puppeteer _evaluateInternal() but with includeCommandLineAPI: true
     //const contextId = context._contextId;
-    const expression = funcString + "\n//# sourceURL=__playwright_evaluation_script__";
+    const expression = funcString + "\n//# sourceURL=__evaluation_script__";
 
     const { exceptionDetails, result } = await cdp
       .send("Runtime.evaluate", {
         expression,
-        //contextId,
+        contextId: cdpContextId,
         returnByValue: true,
         awaitPromise: true,
         userGesture: true,
@@ -230,13 +180,164 @@ export class Browser
       logger.info("Run Script Finished", details, contextName);
     }
 
-    try {
-      await cdp.detach();
-    } catch (e) {
-      logger.warn("Detach failed", details, contextName);
-    }
-
     return result.value;
   }
 }
 
+
+// ==================================================================
+export class Browser extends BaseBrowser
+{
+  constructor() {
+    super();
+    this.browser = null;
+
+    this.firstCDP = null;
+  }
+
+  isLaunched() {
+    if (this.browser) {
+      logger.warn("Context already inited", {}, "browser");
+      return true;
+    }
+
+    return false;
+  }
+
+  async close() {
+    if (this.browser) {
+      await this.browser.close();
+      this.browser = null;
+    }
+  }
+
+  addInitScript(page, script) {
+    return page.evaluateOnNewDocument(script);
+  }
+
+  async _init(launchOpts) {
+    this.browser = await puppeteer.launch(launchOpts);
+
+    const target = this.browser.target();
+
+    this.firstCDP = await target.createCDPSession();
+  }
+
+  numPages() {
+    return this.browser ? this.browser.pages().length : 0;
+  }
+
+  async newWindowPageWithCDP() {
+    // unique url to detect new pages
+    const startPage = "about:blank?_browsertrix" + Math.random().toString(36).slice(2);
+
+    const p = new Promise((resolve) => {
+      const listener = (target) => {
+        if (target.url() === startPage) {
+          resolve(target);
+          this.browser.removeListener("targetcreated", listener);
+        }
+      };
+
+      this.browser.on("targetcreated", listener);
+    });
+
+    try {
+      await this.firstCDP.send("Target.createTarget", {url: startPage, newWindow: true});
+    } catch (e) {
+      const target = this.browser.target();
+
+      this.firstCDP = await target.createCDPSession();
+
+      await this.firstCDP.send("Target.createTarget", {url: startPage, newWindow: true});
+    }
+
+    const target = await p;
+
+    const page = await target.page();
+
+    const device = this.emulateDevice;
+
+    if (device) {
+      if (device.viewport && device.userAgent) {
+        await page.emulate(device);
+      } else if (device.userAgent) {
+        await page.setUserAgent(device.userAgent);
+      }
+    }
+
+    const cdp = await target.createCDPSession();
+
+    return {page, cdp};
+  }
+
+  async responseHeader(resp, header) {
+    return await resp.headers()[header];
+  }
+
+  async evaluateWithCLI(_, frame, cdp, funcString, logData, contextName) {
+    const context = await frame.executionContext();
+    cdp = context._client;
+    const cdpContextId = context._contextId;
+    return await this.evaluateWithCLI_(cdp, frame, cdpContextId, funcString, logData, contextName);
+  }
+
+  interceptRequest(page, callback) {
+    page.on("request", callback);
+  }
+
+  async waitForNetworkIdle(page, params) {
+    return await page.waitForNetworkIdle(params);
+  }
+
+  async setViewport(page, params) {
+    await page.setViewport(params);
+  }
+
+  async getCookies(page) {
+    return await page.cookies();
+  }
+
+  async setCookies(page, cookies) {
+    return await page.setCookie(...cookies);
+  }
+}
+
+
+// ==================================================================
+// Default Chromium args from playwright
+export const defaultArgs = [
+  "--disable-field-trial-config", // https://source.chromium.org/chromium/chromium/src/+/main:testing/variations/README.md
+  "--disable-background-networking",
+  "--enable-features=NetworkService,NetworkServiceInProcess",
+  "--disable-background-timer-throttling",
+  "--disable-backgrounding-occluded-windows",
+  "--disable-back-forward-cache", // Avoids surprises like main request not being intercepted during page.goBack().
+  "--disable-breakpad",
+  "--disable-client-side-phishing-detection",
+  "--disable-component-extensions-with-background-pages",
+  "--disable-component-update", // Avoids unneeded network activity after startup.
+  "--no-default-browser-check",
+  "--disable-default-apps",
+  "--disable-dev-shm-usage",
+  "--disable-extensions",
+  // AvoidUnnecessaryBeforeUnloadCheckSync - https://github.com/microsoft/playwright/issues/14047
+  // Translate - https://github.com/microsoft/playwright/issues/16126
+  "--disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync,Translate",
+  "--allow-pre-commit-input",
+  "--disable-hang-monitor",
+  "--disable-ipc-flooding-protection",
+  "--disable-popup-blocking",
+  "--disable-prompt-on-repost",
+  "--disable-renderer-backgrounding",
+  "--disable-sync",
+  "--force-color-profile=srgb",
+  "--metrics-recording-only",
+  "--no-first-run",
+  "--enable-automation",
+  "--password-store=basic",
+  "--use-mock-keychain",
+  // See https://chromium-review.googlesource.com/c/chromium/src/+/2436773
+  "--no-service-autorun",
+  "--export-tagged-pdf"
+];

util/constants.js

@@ -1,6 +1,6 @@
 
 export const HTML_TYPES = ["text/html", "application/xhtml", "application/xhtml+xml"];
-export const WAIT_UNTIL_OPTS = ["load", "domcontentloaded", "networkidle"];
+export const WAIT_UNTIL_OPTS = ["load", "domcontentloaded", "networkidle0", "networkidle2"];
 export const BEHAVIOR_LOG_FUNC = "__bx_log";
 export const MAX_DEPTH = 1000000;
 

util/originoverride.js

@@ -12,18 +12,25 @@ export class OriginOverride
     });
   }
 
-  initPage(page) {
+  async initPage(browser, page) {
-    for (const {orig, dest} of this.originOverride) {
+    const onRequest = async (request) => {
-      const logDetails = {page: page.url(), orig, dest};
-
-      logger.debug(`Adding override ${orig} => ${dest}`);
-
-      page.route(orig + "/**", async (route) => {
       try {
-          const request = route.request();
         const url = request.url();
 
-          const newUrl = dest + url.slice(orig.length);
+        let newUrl = null;
+
+        for (const {orig, dest} of this.originOverride) {
+          if (url.startsWith(orig)) {
+            newUrl = dest + url.slice(orig.length);
+            break;
+          }
+        }
+
+        if (!newUrl) {
+          request.continue({}, -1);
+          return;
+        }
+
         const resp = await fetch(newUrl, {headers: request.headers()});
 
         const body = Buffer.from(await resp.arrayBuffer());
@@ -32,12 +39,13 @@ export class OriginOverride
 
         logger.debug("Origin overridden", {orig: url, dest: newUrl, status, body: body.length}, "originoverride");
 
-          route.fulfill({body, headers, status});
+        request.respond({body, headers, status}, -1);
 
       } catch (e) {
-          logger.warn("Error overriding origin", {...errJSON(e), ...logDetails}, "originoverride");
+        logger.warn("Error overriding origin", {...errJSON(e), url: page.url()}, "originoverride");
-        }
+        request.continue({}, -1);
-      });
       }
+    };
+    await browser.interceptRequest(page, onRequest);
   }
 }

util/screenshots.js

@@ -28,7 +28,8 @@ export const screenshotTypes = {
 
 export class Screenshots {
 
-  constructor({page, url, date, directory}) {
+  constructor({browser, page, url, date, directory}) {
+    this.browser = browser;
     this.page = page;
     this.url = url;
     this.directory = directory;
@@ -39,7 +40,7 @@ export class Screenshots {
   async take(screenshotType="view") {
     try {
       if (screenshotType !== "fullPage") {
-        await this.page.setViewportSize({width: 1920, height: 1080});
+        await this.browser.setViewport(this.page, {width: 1920, height: 1080});
       }
       const options = screenshotTypes[screenshotType];
       const screenshotBuffer = await this.page.screenshot(options);

yarn.lock

@@ -620,6 +620,20 @@
   resolved "https://registry.yarnpkg.com/@novnc/novnc/-/novnc-1.4.0.tgz#68adae81a741624142b518323441e852c1f34281"
   integrity sha512-kW6ALMc5BuH08e/ond/I1naYcfjc19JYMN1EdtmgjjjzPGCjW8fMtVM3MwM6q7YLRjPlQ3orEvoKMgSS7RkEVQ==
 
+"@puppeteer/browsers@0.5.0":
+  version "0.5.0"
+  resolved "https://registry.yarnpkg.com/@puppeteer/browsers/-/browsers-0.5.0.tgz#1a1ee454b84a986b937ca2d93146f25a3fe8b670"
+  integrity sha512-Uw6oB7VvmPRLE4iKsjuOh8zgDabhNX67dzo8U/BB0f9527qx+4eeUs+korU98OhG5C4ubg7ufBgVi63XYwS6TQ==
+  dependencies:
+    debug "4.3.4"
+    extract-zip "2.0.1"
+    https-proxy-agent "5.0.1"
+    progress "2.0.3"
+    proxy-from-env "1.1.0"
+    tar-fs "2.1.1"
+    unbzip2-stream "1.4.3"
+    yargs "17.7.1"
+
 "@sinclair/typebox@^0.24.1":
   version "0.24.50"
   resolved "https://registry.yarnpkg.com/@sinclair/typebox/-/typebox-0.24.50.tgz#35ee4db4ab8f3a8ff56490c51f92445d2776451e"
@@ -766,6 +780,13 @@
   dependencies:
     "@types/yargs-parser" "*"
 
+"@types/yauzl@^2.9.1":
+  version "2.10.0"
+  resolved "https://registry.yarnpkg.com/@types/yauzl/-/yauzl-2.10.0.tgz#b3248295276cf8c6f153ebe6a9aba0c988cb2599"
+  integrity sha512-Cn6WYCm0tXv8p6k+A8PvbDG763EDpBoTzHdA+Q/MF6H3sapGjCm9NzoaJncJS9tUKSuCoDs9XHxYYsQDgxR6kw==
+  dependencies:
+    "@types/node" "*"
+
 "@zxing/text-encoding@0.9.0":
   version "0.9.0"
   resolved "https://registry.yarnpkg.com/@zxing/text-encoding/-/text-encoding-0.9.0.tgz#fb50ffabc6c7c66a0c96b4c03e3d9be74864b70b"
@@ -781,6 +802,13 @@ acorn@^7.4.0:
   resolved "https://registry.yarnpkg.com/acorn/-/acorn-7.4.1.tgz#feaed255973d2e77555b83dbc08851a6c63520fa"
   integrity sha512-nQyp0o1/mNdbTO1PO6kHkwSrmgZ0MT/jCCpNiwbUjGoRN4dlBhqJtoQuCnEOKzgTVwg0ZWiCoQy6SxMebQVh8A==
 
+agent-base@6:
+  version "6.0.2"
+  resolved "https://registry.yarnpkg.com/agent-base/-/agent-base-6.0.2.tgz#49fff58577cfee3f37176feab4c22e00f86d7f77"
+  integrity sha512-RZNwNclF7+MS/8bDg70amg32dyeZGZxiDuQmZxKLAlQjr3jGyLx+4Kkk58UO7D2QdgFIQCovuSuZESne6RG6XQ==
+  dependencies:
+    debug "4"
+
 ajv@^6.10.0, ajv@^6.12.4:
   version "6.12.6"
   resolved "https://registry.yarnpkg.com/ajv/-/ajv-6.12.6.tgz#baf5a62e802b07d977034586f8c3baf5adf26df4"
@@ -973,6 +1001,20 @@ balanced-match@^1.0.0:
   resolved "https://registry.yarnpkg.com/balanced-match/-/balanced-match-1.0.2.tgz#e83e3a7e3f300b34cb9d87f615fa0cbf357690ee"
   integrity sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==
 
+base64-js@^1.3.1:
+  version "1.5.1"
+  resolved "https://registry.yarnpkg.com/base64-js/-/base64-js-1.5.1.tgz#1b1b440160a5bf7ad40b650f095963481903930a"
+  integrity sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==
+
+bl@^4.0.3:
+  version "4.1.0"
+  resolved "https://registry.yarnpkg.com/bl/-/bl-4.1.0.tgz#451535264182bec2fbbc83a62ab98cf11d9f7b3a"
+  integrity sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==
+  dependencies:
+    buffer "^5.5.0"
+    inherits "^2.0.4"
+    readable-stream "^3.4.0"
+
 block-stream2@^2.0.0:
   version "2.1.0"
   resolved "https://registry.yarnpkg.com/block-stream2/-/block-stream2-2.1.0.tgz#ac0c5ef4298b3857796e05be8ebed72196fa054b"
@@ -1091,6 +1133,11 @@ bser@2.1.1:
   dependencies:
     node-int64 "^0.4.0"
 
+buffer-crc32@~0.2.3:
+  version "0.2.13"
+  resolved "https://registry.yarnpkg.com/buffer-crc32/-/buffer-crc32-0.2.13.tgz#0d333e3f00eac50aa1454abd30ef8c2a5d9a7242"
+  integrity sha512-VO9Ht/+p3SN7SKWqcrgEzjGbRSJYTx+Q1pTQC0wrWqHx0vpJraQ6GtHx8tvcg1rlK1byhU5gccxgOgj7B0TDkQ==
+
 buffer-from@^1.0.0:
   version "1.1.1"
   resolved "https://registry.yarnpkg.com/buffer-from/-/buffer-from-1.1.1.tgz#32713bc028f75c02fdb710d7c7bcec1f2c6070ef"
@@ -1101,6 +1148,14 @@ buffer-xor@^1.0.3:
   resolved "https://registry.yarnpkg.com/buffer-xor/-/buffer-xor-1.0.3.tgz#26e61ed1422fb70dd42e6e36729ed51d855fe8d9"
   integrity sha1-JuYe0UIvtw3ULm42cp7VHYVf6Nk=
 
+buffer@^5.2.1, buffer@^5.5.0:
+  version "5.7.1"
+  resolved "https://registry.yarnpkg.com/buffer/-/buffer-5.7.1.tgz#ba62e7c13133053582197160851a8f648e99eed0"
+  integrity sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==
+  dependencies:
+    base64-js "^1.3.1"
+    ieee754 "^1.1.13"
+
 cacheable-lookup@^5.0.3:
   version "5.0.4"
   resolved "https://registry.yarnpkg.com/cacheable-lookup/-/cacheable-lookup-5.0.4.tgz#5a6b865b2c44357be3d5ebc2a467b032719a7005"
@@ -1174,6 +1229,18 @@ charenc@0.0.2:
   resolved "https://registry.yarnpkg.com/charenc/-/charenc-0.0.2.tgz#c0a1d2f3a7092e03774bfa83f14c0fc5790a8667"
   integrity sha1-wKHS86cJLgN3S/qD8UwPxXkKhmc=
 
+chownr@^1.1.1:
+  version "1.1.4"
+  resolved "https://registry.yarnpkg.com/chownr/-/chownr-1.1.4.tgz#6fc9d7b42d32a583596337666e7d08084da2cc6b"
+  integrity sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==
+
+chromium-bidi@0.4.7:
+  version "0.4.7"
+  resolved "https://registry.yarnpkg.com/chromium-bidi/-/chromium-bidi-0.4.7.tgz#4c022c2b0fb1d1c9b571fadf373042160e71d236"
+  integrity sha512-6+mJuFXwTMU6I3vYLs6IL8A1DyQTPjCfIL971X0aMPVGRbGnNfl6i6Cl0NMbxi2bRYLGESt9T2ZIMRM5PAEcIQ==
+  dependencies:
+    mitt "3.0.0"
+
 ci-info@^3.2.0:
   version "3.5.0"
   resolved "https://registry.yarnpkg.com/ci-info/-/ci-info-3.5.0.tgz#bfac2a29263de4c829d806b1ab478e35091e171f"
@@ -1308,6 +1375,13 @@ create-hmac@^1.1.0, create-hmac@^1.1.4, create-hmac@^1.1.7:
     safe-buffer "^5.0.1"
     sha.js "^2.4.8"
 
+cross-fetch@3.1.5:
+  version "3.1.5"
+  resolved "https://registry.yarnpkg.com/cross-fetch/-/cross-fetch-3.1.5.tgz#e1389f44d9e7ba767907f7af8454787952ab534f"
+  integrity sha512-lvb1SBsI0Z7GDwmuid+mU3kWVBwTVUbe7S0H52yaaAdQOXq2YktTCZdlAcNKFzE6QtRz0snpw9bNiPeOIkkQvw==
+  dependencies:
+    node-fetch "2.6.7"
+
 cross-spawn@^7.0.2, cross-spawn@^7.0.3:
   version "7.0.3"
   resolved "https://registry.yarnpkg.com/cross-spawn/-/cross-spawn-7.0.3.tgz#f73a85b9d5d41d045551c177e2882d4ac85728a6"
@@ -1339,6 +1413,13 @@ crypto-browserify@^3.12.0:
     randombytes "^2.0.0"
     randomfill "^1.0.3"
 
+debug@4, debug@4.3.4:
+  version "4.3.4"
+  resolved "https://registry.yarnpkg.com/debug/-/debug-4.3.4.tgz#1319f6579357f2338d3337d2cdd4914bb5dcc865"
+  integrity sha512-PRWFHuSU3eDtQJPvnNY7Jcket1j0t5OuOsFzPPzsekD52Zl8qUfFIPEiswXqIvHWGVHOgX+7G/vCNNhehwxfkQ==
+  dependencies:
+    ms "2.1.2"
+
 debug@^4.0.1, debug@^4.1.0, debug@^4.1.1, debug@^4.3.1:
   version "4.3.1"
   resolved "https://registry.yarnpkg.com/debug/-/debug-4.3.1.tgz#f0d229c505e0c6d8c49ac553d1b13dc183f6b2ee"
@@ -1403,6 +1484,11 @@ detect-newline@^3.0.0:
   resolved "https://registry.yarnpkg.com/detect-newline/-/detect-newline-3.1.0.tgz#576f5dfc63ae1a192ff192d8ad3af6308991b651"
   integrity sha512-TLz+x/vEXm/Y7P7wn1EJFNLxYpUD4TgMosxY6fAVJUnJMbupHBOncxyWUG9OpTaH9EBD7uFI5LfEgmMOc54DsA==
 
+devtools-protocol@0.0.1107588:
+  version "0.0.1107588"
+  resolved "https://registry.yarnpkg.com/devtools-protocol/-/devtools-protocol-0.0.1107588.tgz#f8cac707840b97cc30b029359341bcbbb0ad8ffa"
+  integrity sha512-yIR+pG9x65Xko7bErCUSQaDLrO/P1p3JUzEk7JCU4DowPcGHkTGUGQapcfcLc4qj0UaALwZ+cr0riFgiqpixcg==
+
 diff-sequences@^29.2.0:
   version "29.2.0"
   resolved "https://registry.yarnpkg.com/diff-sequences/-/diff-sequences-29.2.0.tgz#4c55b5b40706c7b5d2c5c75999a50c56d214e8f6"
@@ -1459,7 +1545,7 @@ emoji-regex@^8.0.0:
   resolved "https://registry.yarnpkg.com/emoji-regex/-/emoji-regex-8.0.0.tgz#e818fd69ce5ccfcb404594f842963bf53164cc37"
   integrity sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==
 
-end-of-stream@^1.1.0:
+end-of-stream@^1.1.0, end-of-stream@^1.4.1:
   version "1.4.4"
   resolved "https://registry.yarnpkg.com/end-of-stream/-/end-of-stream-1.4.4.tgz#5ae64a5f45057baf3626ec14da0ca5e4b2431eb0"
   integrity sha512-+uw1inIHVPQoaVuHzRyXd21icM+cnt4CzD5rW+NC1wjOUSTOs+Te7FOv7AhN7vS9x/oIyhLP5PR1H+phQAHu5Q==
@@ -1725,6 +1811,17 @@ expect@^29.2.1:
     jest-message-util "^29.2.1"
     jest-util "^29.2.1"
 
+extract-zip@2.0.1:
+  version "2.0.1"
+  resolved "https://registry.yarnpkg.com/extract-zip/-/extract-zip-2.0.1.tgz#663dca56fe46df890d5f131ef4a06d22bb8ba13a"
+  integrity sha512-GDhU9ntwuKyGXdZBUgTIe+vXnWj0fppUEtMDL0+idd5Sta8TGpHssn/eusA9mrPr9qNDym6SxAYZjNvCn/9RBg==
+  dependencies:
+    debug "^4.1.1"
+    get-stream "^5.1.0"
+    yauzl "^2.10.0"
+  optionalDependencies:
+    "@types/yauzl" "^2.9.1"
+
 fast-deep-equal@^3.1.1:
   version "3.1.3"
   resolved "https://registry.yarnpkg.com/fast-deep-equal/-/fast-deep-equal-3.1.3.tgz#3a7d56b559d6cbc3eb512325244e619a65c6c525"
@@ -1752,6 +1849,13 @@ fb-watchman@^2.0.0:
   dependencies:
     bser "2.1.1"
 
+fd-slicer@~1.1.0:
+  version "1.1.0"
+  resolved "https://registry.yarnpkg.com/fd-slicer/-/fd-slicer-1.1.0.tgz#25c7c89cb1f9077f8891bbe61d8f390eae256f1e"
+  integrity sha512-cE1qsB/VwyQozZ+q1dGxR8LBYNZeofhEdUNGSMbQD3Gw2lAzX9Zb3uIU6Ebc/Fmyjo9AWWfnn0AUCHqtevs/8g==
+  dependencies:
+    pend "~1.2.0"
+
 file-entry-cache@^6.0.1:
   version "6.0.1"
   resolved "https://registry.yarnpkg.com/file-entry-cache/-/file-entry-cache-6.0.1.tgz#211b2dd9659cb0394b073e7323ac3c933d522027"
@@ -1792,6 +1896,11 @@ foreach@^2.0.5:
   resolved "https://registry.yarnpkg.com/foreach/-/foreach-2.0.5.tgz#0bee005018aeb260d0a3af3ae658dd0136ec1b99"
   integrity sha1-C+4AUBiusmDQo6865ljdATbsG5k=
 
+fs-constants@^1.0.0:
+  version "1.0.0"
+  resolved "https://registry.yarnpkg.com/fs-constants/-/fs-constants-1.0.0.tgz#6be0de9be998ce16af8afc24497b9ee9b7ccd9ad"
+  integrity sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==
+
 fs.realpath@^1.0.0:
   version "1.0.0"
   resolved "https://registry.yarnpkg.com/fs.realpath/-/fs.realpath-1.0.0.tgz#1504ad2523158caa40db4a2787cb01411994ea4f"
@@ -2011,11 +2120,24 @@ http2-wrapper@^1.0.0-beta.5.2:
     quick-lru "^5.1.1"
     resolve-alpn "^1.0.0"
 
+https-proxy-agent@5.0.1:
+  version "5.0.1"
+  resolved "https://registry.yarnpkg.com/https-proxy-agent/-/https-proxy-agent-5.0.1.tgz#c59ef224a04fe8b754f3db0063a25ea30d0005d6"
+  integrity sha512-dFcAjpTQFgoLMzC2VwU+C/CbS7uRL0lWmxDITmqm7C+7F0Odmj6s9l6alZc6AELXhrnggM2CeWSXHGOdX2YtwA==
+  dependencies:
+    agent-base "6"
+    debug "4"
+
 human-signals@^2.1.0:
   version "2.1.0"
   resolved "https://registry.yarnpkg.com/human-signals/-/human-signals-2.1.0.tgz#dc91fcba42e4d06e4abaed33b3e7a3c02f514ea0"
   integrity sha512-B4FFZ6q/T2jhhksgkbEW3HBvWIfDW85snkQgawt07S7J5QXTk6BkNV+0yAeZrM5QpMAdYlocGoljn0sJ/WQkFw==
 
+ieee754@^1.1.13:
+  version "1.2.1"
+  resolved "https://registry.yarnpkg.com/ieee754/-/ieee754-1.2.1.tgz#8eb7a10a63fff25d15a57b001586d177d1b0d352"
+  integrity sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==
+
 ignore@^4.0.6:
   version "4.0.6"
   resolved "https://registry.yarnpkg.com/ignore/-/ignore-4.0.6.tgz#750e3db5862087b4737ebac8207ffd1ef27b25fc"
@@ -2937,6 +3059,16 @@ minio@7.0.26:
     xml "^1.0.0"
     xml2js "^0.4.15"
 
+mitt@3.0.0:
+  version "3.0.0"
+  resolved "https://registry.yarnpkg.com/mitt/-/mitt-3.0.0.tgz#69ef9bd5c80ff6f57473e8d89326d01c414be0bd"
+  integrity sha512-7dX2/10ITVyqh4aOSVI9gdape+t9l2/8QxHrFmUXu4EEUpdlxl6RudZUPZoc+zuY2hk1j7XxVroIVIan/pD/SQ==
+
+mkdirp-classic@^0.5.2:
+  version "0.5.3"
+  resolved "https://registry.yarnpkg.com/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz#fa10c9115cc6d8865be221ba47ee9bed78601113"
+  integrity sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==
+
 mkdirp@^0.5.1:
   version "0.5.5"
   resolved "https://registry.yarnpkg.com/mkdirp/-/mkdirp-0.5.5.tgz#d91cefd62d1436ca0f41620e251288d420099def"
@@ -2954,6 +3086,13 @@ natural-compare@^1.4.0:
   resolved "https://registry.yarnpkg.com/natural-compare/-/natural-compare-1.4.0.tgz#4abebfeed7541f2c27acfb29bdbbd15c8d5ba4f7"
   integrity sha1-Sr6/7tdUHywnrPspvbvRXI1bpPc=
 
+node-fetch@2.6.7:
+  version "2.6.7"
+  resolved "https://registry.yarnpkg.com/node-fetch/-/node-fetch-2.6.7.tgz#24de9fba827e3b4ae44dc8b20256a379160052ad"
+  integrity sha512-ZjMPFEfVx5j+y2yF35Kzx5sF7kDzxuDj6ziH4FFbOp87zKDZNx8yExJIb05OGF4Nlt9IHFIMBkRl41VdvcNdbQ==
+  dependencies:
+    whatwg-url "^5.0.0"
+
 node-int64@^0.4.0:
   version "0.4.0"
   resolved "https://registry.yarnpkg.com/node-int64/-/node-int64-0.4.0.tgz#87a9065cdb355d3182d8f94ce11188b825c68a3b"
@@ -3172,6 +3311,11 @@ pbkdf2@^3.0.3:
     safe-buffer "^5.0.1"
     sha.js "^2.4.8"
 
+pend@~1.2.0:
+  version "1.2.0"
+  resolved "https://registry.yarnpkg.com/pend/-/pend-1.2.0.tgz#7a57eb550a6783f9115331fcf4663d5c8e007a50"
+  integrity sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg==
+
 picocolors@^1.0.0:
   version "1.0.0"
   resolved "https://registry.yarnpkg.com/picocolors/-/picocolors-1.0.0.tgz#cb5bdc74ff3f51892236eaf79d68bc44564ab81c"
@@ -3199,11 +3343,6 @@ pkg-dir@^4.2.0:
   dependencies:
     find-up "^4.0.0"
 
-playwright-core@^1.31.2:
-  version "1.31.2"
-  resolved "https://registry.yarnpkg.com/playwright-core/-/playwright-core-1.31.2.tgz#debf4b215d14cb619adb7e511c164d068075b2ed"
-  integrity sha512-a1dFgCNQw4vCsG7bnojZjDnPewZcw7tZUNFN0ZkcLYKj+mPmXvg4MpaaKZ5SgqPsOmqIf2YsVRkgqiRDxD+fDQ==
-
 prelude-ls@^1.2.1:
   version "1.2.1"
   resolved "https://registry.yarnpkg.com/prelude-ls/-/prelude-ls-1.2.1.tgz#debc6489d7a6e6b0e7611888cec880337d316396"
@@ -3218,7 +3357,7 @@ pretty-format@^29.2.1:
     ansi-styles "^5.0.0"
     react-is "^18.0.0"
 
-progress@^2.0.0:
+progress@2.0.3, progress@^2.0.0:
   version "2.0.3"
   resolved "https://registry.yarnpkg.com/progress/-/progress-2.0.3.tgz#7e8cf8d8f5b8f239c1bc68beb4eb78567d572ef8"
   integrity sha512-7PiHtLll5LdnKIMw100I+8xJXR5gW2QwWYkT6iJva0bXitZKa/XMrSbdmg3r2Xnaidz9Qumd0VPaMrZlF9V9sA==
@@ -3240,6 +3379,11 @@ prop-types@^15.7.2:
     object-assign "^4.1.1"
     react-is "^16.8.1"
 
+proxy-from-env@1.1.0:
+  version "1.1.0"
+  resolved "https://registry.yarnpkg.com/proxy-from-env/-/proxy-from-env-1.1.0.tgz#e102f16ca355424865755d2c9e8ea4f24d58c3e2"
+  integrity sha512-D+zkORCbA9f1tdWRK0RaCR3GPv50cMxcrz4X8k5LTSUD1Dkw47mKJEZQNunItRTkWwgtaUSo1RVFRIG9ZXiFYg==
+
 public-encrypt@^4.0.0:
   version "4.0.3"
   resolved "https://registry.yarnpkg.com/public-encrypt/-/public-encrypt-4.0.3.tgz#4fcc9d77a07e48ba7527e7cbe0de33d0701331e0"
@@ -3265,6 +3409,23 @@ punycode@^2.1.0:
   resolved "https://registry.yarnpkg.com/punycode/-/punycode-2.1.1.tgz#b58b010ac40c22c5657616c8d2c2c02c7bf479ec"
   integrity sha512-XRsRjdf+j5ml+y/6GKHPZbrF/8p2Yga0JPtdqTIY2Xe5ohJPD9saDJJLPvp9+NSBprVvevdXZybnj2cv8OEd0A==
 
+puppeteer-core@^19.11.1:
+  version "19.11.1"
+  resolved "https://registry.yarnpkg.com/puppeteer-core/-/puppeteer-core-19.11.1.tgz#4c63d7a0a6cd268ff054ebcac315b646eee32667"
+  integrity sha512-qcuC2Uf0Fwdj9wNtaTZ2OvYRraXpAK+puwwVW8ofOhOgLPZyz1c68tsorfIZyCUOpyBisjr+xByu7BMbEYMepA==
+  dependencies:
+    "@puppeteer/browsers" "0.5.0"
+    chromium-bidi "0.4.7"
+    cross-fetch "3.1.5"
+    debug "4.3.4"
+    devtools-protocol "0.0.1107588"
+    extract-zip "2.0.1"
+    https-proxy-agent "5.0.1"
+    proxy-from-env "1.1.0"
+    tar-fs "2.1.1"
+    unbzip2-stream "1.4.3"
+    ws "8.13.0"
+
 querystring@0.2.0:
   version "0.2.0"
   resolved "https://registry.yarnpkg.com/querystring/-/querystring-0.2.0.tgz#b209849203bb25df820da756e747005878521620"
@@ -3309,6 +3470,15 @@ react-is@^18.0.0:
     string_decoder "^1.1.1"
     util-deprecate "^1.0.1"
 
+readable-stream@^3.1.1:
+  version "3.6.2"
+  resolved "https://registry.yarnpkg.com/readable-stream/-/readable-stream-3.6.2.tgz#56a9b36ea965c00c5a93ef31eb111a0f11056967"
+  integrity sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==
+  dependencies:
+    inherits "^2.0.3"
+    string_decoder "^1.1.1"
+    util-deprecate "^1.0.1"
+
 redis-commands@1.7.0:
   version "1.7.0"
   resolved "https://registry.yarnpkg.com/redis-commands/-/redis-commands-1.7.0.tgz#15a6fea2d58281e27b1cd1acfb4b293e278c3a89"
@@ -3684,6 +3854,27 @@ table@^6.0.4:
     string-width "^4.2.0"
     strip-ansi "^6.0.0"
 
+tar-fs@2.1.1:
+  version "2.1.1"
+  resolved "https://registry.yarnpkg.com/tar-fs/-/tar-fs-2.1.1.tgz#489a15ab85f1f0befabb370b7de4f9eb5cbe8784"
+  integrity sha512-V0r2Y9scmbDRLCNex/+hYzvp/zyYjvFbHPNgVTKfQvVrb6guiE/fxP+XblDNR011utopbkex2nM4dHNV6GDsng==
+  dependencies:
+    chownr "^1.1.1"
+    mkdirp-classic "^0.5.2"
+    pump "^3.0.0"
+    tar-stream "^2.1.4"
+
+tar-stream@^2.1.4:
+  version "2.2.0"
+  resolved "https://registry.yarnpkg.com/tar-stream/-/tar-stream-2.2.0.tgz#acad84c284136b060dc3faa64474aa9aebd77287"
+  integrity sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==
+  dependencies:
+    bl "^4.0.3"
+    end-of-stream "^1.4.1"
+    fs-constants "^1.0.0"
+    inherits "^2.0.3"
+    readable-stream "^3.1.1"
+
 test-exclude@^6.0.0:
   version "6.0.0"
   resolved "https://registry.yarnpkg.com/test-exclude/-/test-exclude-6.0.0.tgz#04a8698661d805ea6fa293b6cb9e63ac044ef15e"
@@ -3706,6 +3897,11 @@ through2@^3.0.1:
     inherits "^2.0.4"
     readable-stream "2 || 3"
 
+through@^2.3.8:
+  version "2.3.8"
+  resolved "https://registry.yarnpkg.com/through/-/through-2.3.8.tgz#0dd4c9ffaabc357960b1b724115d7e0e86a2e1f5"
+  integrity sha512-w89qg7PI8wAdvX60bMDP+bFoD5Dvhm9oLheFp5O4a2QF0cSBGsBX4qZmadPMvVqlLJBBci+WqGGOAPvcDeNSVg==
+
 tmpl@1.0.5:
   version "1.0.5"
   resolved "https://registry.yarnpkg.com/tmpl/-/tmpl-1.0.5.tgz#8683e0b902bb9c20c4f726e3c0b69f36518c07cc"
@@ -3723,6 +3919,11 @@ to-regex-range@^5.0.1:
   dependencies:
     is-number "^7.0.0"
 
+tr46@~0.0.3:
+  version "0.0.3"
+  resolved "https://registry.yarnpkg.com/tr46/-/tr46-0.0.3.tgz#8184fd347dac9cdc185992f3a6622e14b9d9ab6a"
+  integrity sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw==
+
 type-check@^0.4.0, type-check@~0.4.0:
   version "0.4.0"
   resolved "https://registry.yarnpkg.com/type-check/-/type-check-0.4.0.tgz#07b8203bfa7056c0657050e3ccd2c37730bab8f1"
@@ -3760,6 +3961,14 @@ unbox-primitive@^1.0.0, unbox-primitive@^1.0.1:
     has-symbols "^1.0.2"
     which-boxed-primitive "^1.0.2"
 
+unbzip2-stream@1.4.3:
+  version "1.4.3"
+  resolved "https://registry.yarnpkg.com/unbzip2-stream/-/unbzip2-stream-1.4.3.tgz#b0da04c4371311df771cdc215e87f2130991ace7"
+  integrity sha512-mlExGW4w71ebDJviH16lQLtZS32VKqsSfk80GCfUlwT/4/hNRFsoscrF/c++9xinkMzECL1uL9DDwXqFWkruPg==
+  dependencies:
+    buffer "^5.2.1"
+    through "^2.3.8"
+
 update-browserslist-db@^1.0.9:
   version "1.0.10"
   resolved "https://registry.yarnpkg.com/update-browserslist-db/-/update-browserslist-db-1.0.10.tgz#0f54b876545726f17d00cd9a2561e6dade943ff3"
@@ -3842,6 +4051,19 @@ web-encoding@^1.1.5:
   optionalDependencies:
     "@zxing/text-encoding" "0.9.0"
 
+webidl-conversions@^3.0.0:
+  version "3.0.1"
+  resolved "https://registry.yarnpkg.com/webidl-conversions/-/webidl-conversions-3.0.1.tgz#24534275e2a7bc6be7bc86611cc16ae0a5654871"
+  integrity sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ==
+
+whatwg-url@^5.0.0:
+  version "5.0.0"
+  resolved "https://registry.yarnpkg.com/whatwg-url/-/whatwg-url-5.0.0.tgz#966454e8765462e37644d3626f6742ce8b70965d"
+  integrity sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw==
+  dependencies:
+    tr46 "~0.0.3"
+    webidl-conversions "^3.0.0"
+
 which-boxed-primitive@^1.0.2:
   version "1.0.2"
   resolved "https://registry.yarnpkg.com/which-boxed-primitive/-/which-boxed-primitive-1.0.2.tgz#13757bc89b209b049fe5d86430e21cf40a89a8e6"
@@ -3913,6 +4135,11 @@ write-file-atomic@^4.0.1:
     imurmurhash "^0.1.4"
     signal-exit "^3.0.7"
 
+ws@8.13.0:
+  version "8.13.0"
+  resolved "https://registry.yarnpkg.com/ws/-/ws-8.13.0.tgz#9a9fb92f93cf41512a0735c8f4dd09b8a1211cd0"
+  integrity sha512-x9vcZYTrFPC7aSIbj7sRCYo7L/Xb8Iy+pW0ng0wt2vCJv7M9HOMy0UoN3rr+IFC7hb7vXoqS+P9ktyLLLhO+LA==
+
 ws@^7.4.4:
   version "7.5.9"
   resolved "https://registry.yarnpkg.com/ws/-/ws-7.5.9.tgz#54fa7db29f4c7cec68b1ddd3a89de099942bb591"
@@ -3964,11 +4191,24 @@ yargs-parser@^20.2.2:
   resolved "https://registry.yarnpkg.com/yargs-parser/-/yargs-parser-20.2.9.tgz#2eb7dc3b0289718fc295f362753845c41a0c94ee"
   integrity sha512-y11nGElTIV+CT3Zv9t7VKl+Q3hTQoT9a1Qzezhhl6Rp21gJ/IVTW7Z3y9EWXhuUBC2Shnf+DX0antecpAwSP8w==
 
-yargs-parser@^21.0.0:
+yargs-parser@^21.0.0, yargs-parser@^21.1.1:
   version "21.1.1"
   resolved "https://registry.yarnpkg.com/yargs-parser/-/yargs-parser-21.1.1.tgz#9096bceebf990d21bb31fa9516e0ede294a77d35"
   integrity sha512-tVpsJW7DdjecAiFpbIB1e3qxIQsE6NoPc5/eTdrbbIC4h0LVsWhnoa3g+m2HclBIujHzsxZ4VJVA+GUuc2/LBw==
 
+yargs@17.7.1:
+  version "17.7.1"
+  resolved "https://registry.yarnpkg.com/yargs/-/yargs-17.7.1.tgz#34a77645201d1a8fc5213ace787c220eabbd0967"
+  integrity sha512-cwiTb08Xuv5fqF4AovYacTFNxk62th7LKJ6BL9IGUpTJrWoU7/7WdQGTP2SjKf1dUNBGzDd28p/Yfs/GI6JrLw==
+  dependencies:
+    cliui "^8.0.1"
+    escalade "^3.1.1"
+    get-caller-file "^2.0.5"
+    require-directory "^2.1.1"
+    string-width "^4.2.3"
+    y18n "^5.0.5"
+    yargs-parser "^21.1.1"
+
 yargs@^15.3.1:
   version "15.4.1"
   resolved "https://registry.yarnpkg.com/yargs/-/yargs-15.4.1.tgz#0d87a16de01aee9d8bec2bfbf74f67851730f4f8"
@@ -4012,6 +4252,14 @@ yargs@^17.3.1:
     y18n "^5.0.5"
     yargs-parser "^21.0.0"
 
+yauzl@^2.10.0:
+  version "2.10.0"
+  resolved "https://registry.yarnpkg.com/yauzl/-/yauzl-2.10.0.tgz#c7eb17c93e112cb1086fa6d8e51fb0667b79a5f9"
+  integrity sha512-p4a9I6X6nu6IhoGmBqAcbJy1mlC4j27vEPZX9F4L4/vZT3Lyq1VkFHw/V/PUcB9Buo+DG3iHkT0x3Qya58zc3g==
+  dependencies:
+    buffer-crc32 "~0.2.3"
+    fd-slicer "~1.1.0"
+
 yocto-queue@^0.1.0:
   version "0.1.0"
   resolved "https://registry.yarnpkg.com/yocto-queue/-/yocto-queue-0.1.0.tgz#0294eb3dee05028d31ee1a5fa2c556a6aaf10a1b"