Use VNC for headful profile creation (#197)

* profiles: use vnc for automatic profile creation (fixes #194):
- add x11vnc and serve via vnc when not headless, keep existing screencast for headless mode
- use @novnc/novnc to serve vnc JS library
- add novnc_lite.html to serve the content from an iframe
- optimization: don't show initial blank page / don't wait for initial page in puppeteer

* more vnc work:
- set position of browser at 0,0, avoid needing offset to fit
- add /vncpass endpoint to query vnc password (for use with browsertrix-cloud)
- remove websockify, x11vnc now supports ws connections directly!
- vnc_lite: support reconnecting ws if gracefully disconnected

* x11vnc cleanup: just pass password via cmdline to simplify setup

* make interactive profile creation default, automated enabled only if --automated or --username / --password flags are specified
README updates:
- mention new VNC-based streaming
- mention new --automated flag, move automated info below interactive

* README: adjust auto-login example to use mastodon example instead of twitter, which works more consistently

Dockerfile

@@ -4,7 +4,7 @@ ARG BROWSER_VERSION=105
 FROM ${BROWSER_IMAGE_BASE}:${BROWSER_VERSION}
 
 # TODO: Move this into base image
-RUN apt-get update && apt-get install -y jq
+RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends -qqy jq x11vnc
 
 # needed to add args to main build stage
 ARG BROWSER_VERSION
@@ -17,7 +17,8 @@ ENV PROXY_HOST=localhost \
     GEOMETRY=1360x1020x16 \
     BROWSER_VERSION=${BROWSER_VERSION} \
     BROWSER_BIN=google-chrome \
-    OPENSSL_CONF=/app/openssl.conf
+    OPENSSL_CONF=/app/openssl.conf \
+    VNC_PASS=vncpassw0rd!
 
 WORKDIR /app
 

README.md

@@ -15,7 +15,7 @@ Thus far, Browsertrix Crawler supports:
 - Screencasting: Ability to watch crawling in real-time (experimental).
 - Optimized (non-browser) capture of non-HTML resources.
 - Extensible Puppeteer driver script for customizing behavior per crawl or page.
-- Ability to create and reuse browser profiles with user/password login or via interactive login through an embedded browser.
+- Ability to create and reuse browser profiles interactively or via automated user/password login using an embedded browser.
 - Multi-platform support -- prebuilt Docker images available for Intel/AMD and Apple (M1) CPUs.
 
 ## Getting Started
@@ -563,17 +563,69 @@ When The crawler will keep the last `--saveStateHistory` save states and delete
 Browsertrix Crawler also includes a way to use existing browser profiles when running a crawl. This allows pre-configuring the browser, such as by logging in
 to certain sites or setting other settings, and running a crawl exactly with those settings. By creating a logged in profile, the actual login credentials are not included in the crawl, only (temporary) session cookies.
 
-Browsertrix Crawler includes a script to login to a single website with supplied credentials and then save the profile, as well as a new 'interactive' profile creation mode.
-The script profile creation system also take a screenshot so you can check if the login succeeded. The `--url` parameter should specify the URL of a login page.
 
-For example, to create a profile logged in to Twitter, you can run:
+### Interactive Profile Creation
+
+For creating profiles of more complex sites, or logging in to multiple sites at once, the interactive profile creation mode can be used.
+To use this mode, don't specify --username or --password flags and expose two ports on the Docker container to allow DevTools to connect to the browser and to serve
+a status page.
+
+In profile creation mode, Browsertrix Crawler launches a browser which uses VNC (via noVNC) server running on port 6080 to provide a 'remote desktop' for interacting with the browser.
+
+After interactively logging into desired sites or configuring other settings, the 'Create Profile' should be clicked to initiate profile creation.
+Browsertrix Crawler will then stop the browser, and save the browser profile.
+
+For example, to start in interactive profile creation mode, run:
+
+```
+docker run -p 6080:6080 -p 9223:9223 -v $PWD/crawls/profiles:/crawls/profiles/ -it webrecorder/browsertrix-crawler create-login-profile --url "https://example.com/"
+```
+
+Then, open a browser pointing to `http://localhost:9223/` and use the embedded browser to log in to any sites or configure any settings as needed.
+Click 'Create Profile at the top when done. The profile will then be created in `./crawls/profiles/profile.tar.gz` containing the settings of this browsing session.
+
+It is also possible to extend an existing profiles by also passing in an existing profile via the `--profile` flag. In this way, it is possible to build new profiles by extending previous browsing sessions as needed.
+
+```
+docker run -p 6080:6080 -p 9223:9223 -v $PWD/crawls/profiles:/crawls/profiles -it webrecorder/browsertrix-crawler create-login-profile --url "https://example.com/ --filename /crawls/profiles/newProfile.tar.gz --profile /crawls/profiles/oldProfile.tar.gz"
+```
+
+#### Headless vs Headful Profiles
+
+Browsertrix Crawler supports both 'headful' and headless crawling. We recommend using headful crawling to be most accurate to user experience, however,
+headless crawling may be faster.
+
+To use profiles in headless mode, profiles should also be created with `--headless` flag.
+
+When creating browser profile in headless mode, Browsertrix will use the devtools protocol on port 9222 to stream the browser interface (previously, this was also used
+in headful mode as well).
+
+To create a profile in headless mode, run:
+
+```
+docker run -p 9222:9222 -p 9223:9223 -v $PWD/crawls/profiles:/crawls/profiles/ -it webrecorder/browsertrix-crawler create-login-profile --headless --url "https://example.com/"
+```
+
+### Automated Profile Creation for User Login
+
+If the `--automated` flag is provided, Browsertrix Crawler will attempt to create a profile automatically after logging in to sites with a username and password.
+The username and password can be provided via `--username` and `--password` flags or, if omitted, from a command-line prompt.
+
+When using `--automated` or `--username` / `--password`, Browsertrix Crawler will not launch an interactive browser and instead will attempt to finish automatically.
+
+The automated profile creation system will log in to a single website with supplied credentials and then save the profile
+The script profile creation system also take a screenshot so you can check if the login succeeded.
+
+For example, to launch a browser, and login to the digipres.club Mastodon instance, run:
 
 ```bash
-docker run -v $PWD/crawls/profiles:/crawls/profiles -it webrecorder/browsertrix-crawler create-login-profile --url "https://twitter.com/login"
+docker run -v $PWD/crawls/profiles:/crawls/profiles -it webrecorder/browsertrix-crawler create-login-profile --url "https://digipres.club/"
 ```
 
 The script will then prompt you for login credentials, attempt to login and create a tar.gz file in `./crawls/profiles/profile.tar.gz`.
 
+- The `--url` parameter should specify the URL of a login page.
+
 - To specify a custom filename, pass along `--filename` parameter.
 
 - To specify the username and password on the command line (for automated profile creation), pass a `--username` and `--password` flags.
@@ -583,35 +635,7 @@ The script will then prompt you for login credentials, attempt to login and crea
 - To specify the window size for the profile creation embedded browser, specify `--windowSize WIDTH,HEIGHT`. (The default is 1600x900)
 
 
-The current profile creation script is still experimental and the script attempts to detect the username and password fields on a site as generically as possible, but may not work for all sites. Additional profile functionality, such as support for custom profile creation scripts, may be added in the future.
+The current profile creation script is still experimental and the script attempts to detect the username and password fields on a site as generically as possible, but may not work for all sites. Additional automated profile creation functionality, such as support for custom profile creation scripts, may be added in the future.
-
-
-### Interactive Profile Creation
-
-For creating profiles of more complex sites, or logging in to multiple sites at once, the interactive profile creation mode can be used.
-To use this mode, specify the `--interactive` flag and expose two ports on the Docker container to allow DevTools to connect to the browser and to serve
-a status page.
-
-In this mode, Browsertrix launches a browser connected to DevTools, and allowing the user to use the browser via the devtools device UI.
-
-After interactively logging into desired sites or configuring other settings, the 'Create Profile' should be clicked to initiate profile creation.
-
-Browsertrix Crawler will then create a profile as before using the current state of the browser and disconnect from devtools.
-
-For example, to start in interactive profile creation mode, run:
-
-```
-docker run -p 9222:9222 -p 9223:9223 -v $PWD/crawls/profiles:/crawls/profiles/ -it webrecorder/browsertrix-crawler create-login-profile --interactive --url "https://example.com/"
-```
-
-Then, open a browser pointing to `http://localhost:9223/` and use the embedded browser to log in to any sites or configure any settings as needed.
-Click 'Create Profile at the top when done. The profile will then be created in `./crawls/profiles/profile.tar.gz` containing the settings of this browsing session.
-
-It is also possible to extend an existing profiles by also passing in an existing profile via the `--profile` flag. In this way, it is possible to build new profiles by extending previous browsing sessions as needed.
-
-```
-docker run -p 9222:9222 -p 9223:9223 -v $PWD/crawls/profiles:/crawls/profiles -it webrecorder/browsertrix-crawler create-login-profile --interactive --url "https://example.com/ --filename /crawls/profiles/newProfile.tar.gz --profile /crawls/profiles/oldProfile.tar.gz"
-```
 
 ### Using Browser Profile with a Crawl
 
@@ -621,7 +645,7 @@ After running the above command, you can now run a crawl with the profile, as fo
 
 ```bash
 
-docker run -v $PWD/crawls:/crawls/ -it webrecorder/browsertrix-crawler crawl --profile /crawls/profiles/profile.tar.gz --url https://twitter.com/ --generateWACZ --collection test-with-profile
+docker run -v $PWD/crawls:/crawls/ -it webrecorder/browsertrix-crawler crawl --profile /crawls/profiles/profile.tar.gz --url https://digipres.club/ --generateWACZ --collection test-with-profile
 ```
 
 Profiles can also be loaded from an http/https URL, eg. `--profile https://example.com/path/to/profile.tar.gz`

crawler.js

@@ -296,6 +296,7 @@ export class Crawler {
       args: chromeArgs(!process.env.NO_PROXY, this.userAgent, this.extraChromeArgs()),
       userDataDir: this.profileDir,
       defaultViewport: null,
+      waitForInitialPage: false
     };
   }
 

create-login-profile.js

@@ -14,6 +14,7 @@ import { getBrowserExe, loadProfile, saveProfile, chromeArgs, sleep } from "./ut
 import { initStorage } from "./util/storage.js";
 
 const profileHTML = fs.readFileSync(new URL("html/createProfile.html", import.meta.url), {encoding: "utf8"});
+const vncHTML = fs.readFileSync(new URL("html/vnc_lite.html", import.meta.url), {encoding: "utf8"});
 
 const behaviors = fs.readFileSync(new URL("./node_modules/browsertrix-behaviors/dist/behaviors.js", import.meta.url), {encoding: "utf8"});
 
@@ -48,12 +49,18 @@ function cliOpts() {
       default: false,
     },
 
-    "interactive": {
+    "automated": {
-      describe: "Start in interactive mode!",
+      describe: "Start in automated mode, no interactive browser",
       type: "boolean",
       default: false,
     },
 
+    "interactive": {
+      describe: "Deprecated. Now the default option!",
+      type: "boolean",
+      default: false
+    },
+
     "shutdownWait": {
       describe: "Shutdown browser in interactive after this many seconds, if no pings received",
       type: "number",
@@ -68,7 +75,7 @@ function cliOpts() {
     "windowSize": {
       type: "string",
       describe: "Browser window dimensions, specified as: width,height",
-      default: "1600,900"
+      default: getDefaultWindowSize()
     },
 
     "proxy": {
@@ -84,6 +91,13 @@ function cliOpts() {
   };
 }
 
+function getDefaultWindowSize() {
+  const values = process.env.GEOMETRY.split("x");
+  const x = Number(values[0]);
+  const y = Number(values[1]);
+  return `${x},${y}`;
+}
+
 
 
 async function main() {
@@ -105,6 +119,24 @@ async function main() {
       "+extension",
       "RANDR"
     ]);
+
+    //await fsp.mkdir(path.join(homedir(), ".vnc"), {recursive: true});
+
+    //child_process.spawnSync("x11vnc", ["-storepasswd", process.env.VNC_PASS, path.join(homedir(), ".vnc", "passwd")]);
+
+    child_process.spawn("x11vnc", [
+      "-forever",
+      "-ncache_cr",
+      "-xdamage",
+      "-usepw",
+      "-shared",
+      "-rfbport",
+      "6080",
+      "-passwd",
+      process.env.VNC_PASS,
+      "-display",
+      process.env.DISPLAY
+    ]);
   }
 
   let useProxy = false;
@@ -120,6 +152,7 @@ async function main() {
   }
 
   const browserArgs = chromeArgs(useProxy, null, [
+    "--window-position=0,0",
     `--window-size=${params.windowSize}`,
   ]);
 
@@ -133,13 +166,22 @@ async function main() {
     args: browserArgs,
     userDataDir: profileDir,
     defaultViewport: null,
+    waitForInitialPage: false
   };
 
-  if (!params.user && !params.interactive) {
+  if (params.interactive) {
+    console.log("Note: the '--interactive' flag is now deprecated and is the default profile creation option. Use the --automated flag to specify non-interactive mode");
+  }
+
+  if (params.user || params.password) {
+    params.automated = true;
+  }
+
+  if (!params.user && params.automated) {
     params.user = await promptInput("Enter username: ");
   }
 
-  if (!params.password && !params.interactive) {
+  if (!params.password && params.automated) {
     params.password = await promptInput("Enter password: ", true);
   }
 
@@ -151,26 +193,28 @@ async function main() {
 
   await page.setCacheEnabled(false);
 
-  if (params.interactive) {
+  if (!params.automated) {
     await page.evaluateOnNewDocument("Object.defineProperty(navigator, \"webdriver\", {value: false});");
     // for testing, inject browsertrix-behaviors
     await page.evaluateOnNewDocument(behaviors + ";\nself.__bx_behaviors.init();");
   }
 
-  console.log("loading");
+  console.log(`Loading page: ${params.url}`);
 
   await page.goto(params.url, {waitUntil});
-  
-  console.log("loaded");
 
-  if (params.interactive) {
+  if (!params.automated) {
     new InteractiveBrowser(params, browser, page);
-    return;
+  } else {
+    await automatedProfile(params, browser, page, waitUntil);
   }
+}
 
-
+async function automatedProfile(params, browser, page, waitUntil) {
   let u, p;
 
+  console.log("Looking for username and password entry fields on page...");
+
   try {
     u = await page.waitForXPath("//input[contains(@name, 'user') or contains(@name, 'email')]");
     p = await page.waitForXPath("//input[contains(@name, 'pass') and @type='password']");
@@ -303,6 +347,12 @@ class InteractiveBrowser {
     const port = 9223;
     httpServer.listen(port);
     console.log(`Browser Profile UI Server started. Load http://localhost:${port}/ to interact with a Chromium-based browser, click 'Create Profile' when done.`);
+
+    if (!params.headless) {
+      console.log("Screencasting with VNC on port 6080");
+    } else {
+      console.log("Screencasting with CDP on port 9222");
+    }
   }
 
   handlePageLoad() {
@@ -356,11 +406,21 @@ class InteractiveBrowser {
 
     switch (pathname) {
     case "/":
-      targetUrl = `http://$HOST:9222/devtools/inspector.html?ws=$HOST:9222/devtools/page/${this.targetId}&panel=resources`;
       res.writeHead(200, {"Content-Type": "text/html"});
+      if (this.params.headless) {
+        targetUrl = `http://$HOST:9222/devtools/inspector.html?ws=$HOST:9222/devtools/page/${this.targetId}&panel=resources`;
+      } else {
+        targetUrl = `http://$HOST:9223/vnc/?host=$HOST&port=6080&password=${process.env.VNC_PASS}`;
+      }
       res.end(profileHTML.replace("$DEVTOOLS_SRC", targetUrl.replaceAll("$HOST", parsedUrl.hostname)));
       return;
 
+    case "/vnc/":
+    case "/vnc/index.html":
+      res.writeHead(200, {"Content-Type": "text/html"});
+      res.end(vncHTML);
+      return;
+
     case "/ping":
       if (this.shutdownWait) {
         clearInterval(this.shutdownTimer);
@@ -380,6 +440,11 @@ class InteractiveBrowser {
       res.end(JSON.stringify({targetId: this.targetId}));
       return;
 
+    case "/vncpass":
+      res.writeHead(200, {"Content-Type": "application/json"});
+      res.end(JSON.stringify({password: process.env.VNC_PASS}));
+      return;
+
     case "/navigate":
       if (req.method !== "POST") {
         break;
@@ -448,6 +513,14 @@ class InteractiveBrowser {
       return;
     }
 
+    if (pathname.startsWith("/vnc/")) {
+      const fileUrl = new URL("node_modules/@novnc/novnc/" + pathname.slice("/vnc/".length), import.meta.url);
+      const file = fs.readFileSync(fileUrl, {encoding: "utf-8"});
+      res.writeHead(200, {"Content-Type": "application/javascript"});
+      res.end(file);
+      return;
+    }
+
     res.writeHead(404, {"Content-Type": "text/html"});
     res.end("Not Found");
   }

html/vnc_lite.html

@@ -0,0 +1,195 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+
+    <!--
+    noVNC example: lightweight example using minimal UI and features
+
+    This is a self-contained file which doesn't import WebUtil or external CSS.
+
+    Copyright (C) 2019 The noVNC Authors
+    noVNC is licensed under the MPL 2.0 (see LICENSE.txt)
+    This file is licensed under the 2-Clause BSD license (see LICENSE.txt).
+
+    Connect parameters are provided in query string:
+        http://example.com/?host=HOST&port=PORT&scale=true
+    -->
+    <title>noVNC</title>
+
+    <meta charset="utf-8">
+
+    <style>
+
+        body {
+            margin: 0;
+            background-color: dimgrey;
+            height: 100%;
+            display: flex;
+            flex-direction: column;
+        }
+        html {
+            height: 100%;
+        }
+
+        #top_bar {
+            display: none;
+            background-color: #6e84a3;
+            color: white;
+            font: bold 12px Helvetica;
+            padding: 6px 5px 4px 5px;
+            border-bottom: 1px outset;
+        }
+        #status {
+            text-align: center;
+        }
+        #sendCtrlAltDelButton {
+            display: none;
+            position: fixed;
+            top: 0px;
+            right: 0px;
+            border: 1px outset;
+            padding: 5px 5px 4px 5px;
+            cursor: pointer;
+        }
+
+        #screen {
+            flex: 1; /* fill remaining space */
+            overflow: hidden;
+        }
+
+    </style>
+
+    <script type="module" crossorigin="anonymous">
+        // RFB holds the API to connect and communicate with a VNC server
+        import RFB from './core/rfb.js';
+
+        let rfb;
+        let desktopName;
+
+        // When this function is called we have
+        // successfully connected to a server
+        function connectedToServer(e) {
+            status("Connected to " + desktopName);
+        }
+
+        // This function is called when we are disconnected
+        function disconnectedFromServer(e) {
+            if (e.detail.clean) {
+                status("Disconnected, retrying...");
+                setTimeout(connect, 2000);
+            } else {
+                status("Something went wrong, connection is closed");
+            }
+        }
+
+        // When this function is called, the server requires
+        // credentials to authenticate
+        function credentialsAreRequired(e) {
+            const password = prompt("Password Required:");
+            rfb.sendCredentials({ password: password });
+        }
+
+        // When this function is called we have received
+        // a desktop name from the server
+        function updateDesktopName(e) {
+            desktopName = e.detail.name;
+        }
+
+        // Since most operating systems will catch Ctrl+Alt+Del
+        // before they get a chance to be intercepted by the browser,
+        // we provide a way to emulate this key sequence.
+        function sendCtrlAltDel() {
+            rfb.sendCtrlAltDel();
+            return false;
+        }
+
+        // Show a status text in the top bar
+        function status(text) {
+            document.getElementById('status').textContent = text;
+        }
+
+        // This function extracts the value of one variable from the
+        // query string. If the variable isn't defined in the URL
+        // it returns the default value instead.
+        function readQueryVariable(name, defaultValue) {
+            // A URL with a query parameter can look like this (But will most probably get logged on the http server):
+            // https://www.example.com?myqueryparam=myvalue
+            //
+            // For privacy (Using a hastag #, the parameters will not be sent to the server)
+            // the url can be requested in the following way:
+            // https://www.example.com#myqueryparam=myvalue&password=secreatvalue
+            //
+            // Even Mixing public and non public parameters will work:
+            // https://www.example.com?nonsecretparam=example.com#password=secreatvalue
+            //
+            // Note that we use location.href instead of location.search
+            // because Firefox < 53 has a bug w.r.t location.search
+            const re = new RegExp('.*[?&]' + name + '=([^&#]*)'),
+                  match = ''.concat(document.location.href, window.location.hash).match(re);
+
+            if (match) {
+                // We have to decode the URL since want the cleartext value
+                return decodeURIComponent(match[1]);
+            }
+
+            return defaultValue;
+        }
+
+        document.getElementById('sendCtrlAltDelButton')
+            .onclick = sendCtrlAltDel;
+
+        // Read parameters specified in the URL query string
+        // By default, use the host and port of server that served this file
+        const host = readQueryVariable('host', window.location.hostname);
+        let port = readQueryVariable('port', window.location.port);
+        const password = readQueryVariable('password');
+        const path = readQueryVariable('path', 'websockify');
+
+        // | | |         | | |
+        // | | | Connect | | |
+        // v v v         v v v
+        function connect() {
+          status("Connecting");
+
+          // Build the websocket URL used to connect
+          let url;
+          if (window.location.protocol === "https:") {
+              url = 'wss';
+          } else {
+              url = 'ws';
+          }
+          url += '://' + host;
+          if(port) {
+              url += ':' + port;
+          }
+          url += '/' + path;
+
+          // Creating a new RFB object will start a new connection
+          rfb = new RFB(document.getElementById('screen'), url,
+                        { credentials: { password: password } });
+
+          // Add listeners to important events from the RFB module
+          rfb.addEventListener("connect",  connectedToServer);
+          rfb.addEventListener("disconnect", disconnectedFromServer);
+          rfb.addEventListener("credentialsrequired", credentialsAreRequired);
+          rfb.addEventListener("desktopname", updateDesktopName);
+
+          // Set parameters that can be changed on an active connection
+          rfb.viewOnly = readQueryVariable('view_only', false);
+          rfb.scaleViewport = readQueryVariable('scale', false);
+        }
+
+        connect();
+    </script>
+</head>
+
+<body>
+    <div id="top_bar">
+        <div id="status">Loading</div>
+        <div id="sendCtrlAltDelButton">Send CtrlAltDel</div>
+    </div>
+    <div id="screen">
+        <!-- This is where the remote screen will appear -->
+    </div>
+</body>
+</html>

package.json

@@ -11,6 +11,7 @@
     "test": "yarn node --experimental-vm-modules $(yarn bin jest)"
   },
   "dependencies": {
+    "@novnc/novnc": "1.4.0-beta",
     "abort-controller": "^3.0.0",
     "browsertrix-behaviors": "^0.3.4",
     "get-folder-size": "^4.0.0",

requirements.txt

@@ -1,3 +1,3 @@
-pywb>=2.6.8
+pywb>=2.7.2
 uwsgi
 wacz>=0.4.6

util/browser.js

@@ -108,7 +108,7 @@ const DEFAULT_PLAYWRIGHT_FLAGS = [
   "--force-color-profile=srgb",
   "--metrics-recording-only",
   "--no-first-run",
-  "--enable-automation",
+  "--no-startup-window",
   "--password-store=basic",
   "--use-mock-keychain",
   // See https://chromium-review.googlesource.com/c/chromium/src/+/2436773

yarn.lock

@@ -615,6 +615,11 @@
     "@jridgewell/resolve-uri" "3.1.0"
     "@jridgewell/sourcemap-codec" "1.4.14"
 
+"@novnc/novnc@1.4.0-beta":
+  version "1.4.0-beta"
+  resolved "https://registry.yarnpkg.com/@novnc/novnc/-/novnc-1.4.0-beta.tgz#a9aedc3f0274863dcfd0d382c43615e912f7c006"
+  integrity sha512-iLwlvPucpqZ14yZHIrW6bxeC1Aynd5hNhbe9iSEYTOPtOicpVkbwj5Mpkmyw9rSqYoqwLKerV7OJ8afUg1Yq0g==
+
 "@sinclair/typebox@^0.24.1":
   version "0.24.50"
   resolved "https://registry.yarnpkg.com/@sinclair/typebox/-/typebox-0.24.50.tgz#35ee4db4ab8f3a8ff56490c51f92445d2776451e"