Run the browser tests in CloakBrowser, attached to its vendor's container

[jwrosewell]

What this does

Adds CloakBrowser as a fourth browser the suite can drive, beside Chrome, Firefox and Edge, so that the browser tests of every example can also be run in it. CloakBrowser is a patched Chromium that its vendor publishes for browser automation.

Nothing changes for a run that does not opt in. Every new test is skipped, with a reason that names the variable to set, until CLOAK_DEBUGGER_ADDRESS is set.

How it works

The CloakBrowser binary is closed source, so the suite never downloads it and never starts it. The browser runs only inside the vendor's Docker image, in the vendor's server mode (cloakserve), and the suite attaches a ChromeDriver to it through the DevTools port the container serves.

Variable	Meaning
CLOAK_DEBUGGER_ADDRESS	Host and port of the running browser, for example 127.0.0.1:9222. While it is unset every Cloak test is skipped.
CLOAKWEBDRIVER	Optional. A chromedriver of the browser's major version, or the directory holding one. Selenium Manager fetches one when it is unset.

What was added, by file:

• Helpers/BrowserDrivers.cs
  • CreateCloak(ChromeOptions) reads CLOAK_DEBUGGER_ADDRESS, asks the browser for its version at /json/version, finds a chromedriver of that major version, sets ChromeOptions.DebuggerAddress and attaches.
  • The driver is the one CLOAKWEBDRIVER names, or else the one Selenium Manager fetches for that version. A chromedriver on the path is never used, because it belongs to the machine's own Chrome. FindNamedDriver is the lookup that does not search the path, and FindDriver now calls it before searching the path, with the same result as before.
  • The browser outlives the driver, so each attach moves to a new tab, closes the others and clears the cookies and the cache.
• Helpers/RequiresCloakAttribute.cs is a test condition that skips a test, or a whole class, while CLOAK_DEBUGGER_ADDRESS is unset, giving the variable as the reason.
• Selenium/DriverSmokeTests.cs (category Browser) gains Cloak_RunsThePage and Cloak_StartsEachTestClean.
• Browsers/CloakTests.cs (category CloudInternal) is the counterpart of ChromeTests.
• The four Contract classes each gain a _Cloak test beside the existing ones, being Example_RendersRealDetectionResult_Cloak, Example_ServesCoreJs_AndClientSideOverridesFlow_Cloak, Example_RendersClientSideOverridesOnThePage_Cloak and SessionStorageCache_Cloak. The existing tests keep their names and their assertions. Example_RendersClientSideOverridesOnThePage_Cloak reads the size it expects from the browser, and fails if the browser reports a screen of 0.
• Helpers/BrowserDriversCloakTests.cs holds unit tests for the address, the version, the driver lookup and the skip reason. They start no browser.
• .github/workflows/build-and-test.yml gains a cloak job, which starts the image by digest on ubuntu-latest, runs the Cloak tests of the Browser category attached to it, and fails unless Cloak_RunsThePage passed.
• README.md gains the two variables and a section called Running under CloakBrowser, which also says which keys the tests give the browser.

How a consumer opts in

1. Start the container before the tests. This is the form for a Linux CI runner.

   docker run -d --name cloak --network host \
     -e CLOAKBROWSER_AUTO_UPDATE=false \
     cloakhq/cloakbrowser:0.5.12@sha256:2fdd1289154f30594c7bab74943ea7baa6b617a9ec026ab4bf339c2ebabcb712 \
     cloakserve

2. Set CLOAK_DEBUGGER_ADDRESS=127.0.0.1:9222, and optionally SE_AVOID_BROWSER_DOWNLOAD=true.

3. Run the same dotnet test command as today. The Cloak tests of the chosen category then run beside the Chrome and Firefox ones.

4. Remove the container with docker rm -f cloak.

The cloak job in this repository's workflow is a working copy of those steps. With Docker Desktop on Windows the README gives a second form, which publishes the port on the loopback address and maps localhost to the host.

What was checked and found

Checked on Windows 11 with Docker Desktop (engine 29.7.2), against cloakhq/cloakbrowser:0.5.12 at the digest above, and in a Linux container for the host networking case. Findings 8 to 12 come from running the Contract category against a real example.

1. ChromeDriver attaches through cloakserve. cloakserve is a proxy in front of the browser's own DevTools port. ChromeDriver asks it for /json/version and /json/list and then opens the browser WebSocket, and the proxy passes all three on. ChromeDriver 146.0.7680.165, fetched by Selenium Manager, attached to the browser, which reports Chrome/146.0.7680.177.
2. The driver must be 146, whatever Chrome the machine has. The test machine has Chrome 154 and a chromedriver 151 on the path, and neither is used.
3. Arguments never reach the browser. --headless and --user-agent in the options are ignored, because the container started the browser. It sends its own user agent, Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36, and it runs with a window on the container's virtual display.
4. ChromeDriver refuses mobile emulation for a browser it attaches to, with unrecognized chrome option: mobileEmulation. A device metrics override sent through DevTools after attaching changed devicePixelRatio and left screen.width and screen.height at 1920 and 1080.
5. Cookies and session storage survive from one driver to the next unless they are cleared, because the browser keeps running. A cookie set through one driver was sent on the first request of the next.
6. The container looks for updates when it starts. Its log shows a request to PyPI and one to the GitHub releases API, and the source of the vendor's wrapper shows it downloading a newer browser build in the background when it finds one. CLOAKBROWSER_AUTO_UPDATE=false, the vendor's documented setting, turns that off, and the container log then shows neither request.
7. The license in the image is version 1.3, July 2026. /app/BINARY-LICENSE.md inside the image has the same SHA-256 (a959b6f9db58f7e273694368659140e9d82960d964ab48b5f6cf9c4545cc2981) as the file at the commit the README links. The digest names an image index that covers linux/amd64 and linux/arm64.
8. The Contract tests pass in CloakBrowser against a real example. The example was the cloud web example of the .NET examples, Examples/Cloud/GettingStarted-Web, pointed at https://cloud.51degrees.com/ with a paid resource key. It was started on its own and given to the suite as EXAMPLE_URL, which is how CI calls the suite. With CLOAK_DEBUGGER_ADDRESS set all 14 results passed, and without it 9 passed and 5 were skipped. The five CloakBrowser results passed in eight runs out of eight.
9. The browser's own screen size comes back through the client-side overrides. CloakBrowser reports a screen of 1920 by 1080, on a blank tab and on the example's page alike. The example's server-side rows read 0 for the screen width and height, its client-side rows read 1920 and 1080, and the data given to fod.complete holds the same two numbers. The two tests that emulate no device rest on exactly that.
10. Those passes depend on the script's call back. With the script's POST to /51dpipeline/json refused, four of the five CloakBrowser results fail, each on the assertion about what should have come back. Example_RendersRealDetectionResult_Cloak still passes, because it reads only the page the server renders.
11. The .NET cloud example gives the browser no key. Before any of its pages was loaded in CloakBrowser, everything the Contract tests make a browser load from it was fetched and compared with the resource key, being the page, its stylesheet and scripts, /51Degrees.core.js with and without the cookie parameter, and the call back to /51dpipeline/json. None carried the key. The script also calls https://cloud.51degrees.com/api/v4/3pc from the browser, and that answer carries no key either. Every run then went through a pass-through in front of the example that would have withheld any answer carrying the key, and across more than 390 answers none did. The control run was repeated straight at the example with the same 9 passes, so the pass-through changes no result.
12. CloakTests does give the browser keys. The class fetches the cloud's JavaScript include itself and runs it in the browser. Asked for the way the class asks, the include carries the resource key and the license it was requested with. The builder that makes the include copies the request's query parameters into the script, which is how the license gets there. CloakTests was therefore not run in CloakBrowser here, and the README now says which keys a pipeline hands over. The page-based CloudInternal classes put a resource key in a script address on their pages, and they drive Chrome, Firefox and Edge only.

Where this differs from the first plan, and why

• Skipped by a condition attribute, not by Assert.Inconclusive in the test. The Contract classes start the example in their initialize method, which runs before the test body. An attribute skips the test before that, so no example is started for a test that cannot use it. The reason printed is the same kind of line the existing skips give.
• The two client-side override tests emulate no device in CloakBrowser, because of finding 4. Example_ServesCoreJs_AndClientSideOverridesFlow_Cloak keeps the assertions that the screen size and the device id came back, and drops the two that check the emulation. Example_RendersClientSideOverridesOnThePage_Cloak expects the page to render the screen size the browser reports for itself, and fails if that size is 0, because the server-side rows can read 0 and would then match whether or not the call back ran. Example_RendersRealDetectionResult_Cloak sets no user agent, because of finding 3.
• The version comes from the browser, not from a constant, so moving the image pin needs no code change.
• Selenium Manager is asked for the driver directly, and only the driver path it returns is used. Setting ChromeOptions.BrowserVersion and leaving the rest to Selenium worked on a machine that has a Chrome, and failed with Browser not found in a container with no Chrome when SE_AVOID_BROWSER_DOWNLOAD=true.
• Each attach starts clean, because of finding 5.
• The container is started with CLOAKBROWSER_AUTO_UPDATE=false, because of finding 6.
• The cloak job runs only the Cloak tests of the Browser category. It first ran the whole category, which starts Chrome and Firefox as well, and in one of four runs Firefox took more than 60 seconds to start and failed the job although both Cloak tests had passed. Nothing found points at the container as the cause. With the container beside it Firefox took 3, 19, 31 and more than 60 seconds, and the browsers job, which has no container, took 46, 48 and 57 seconds for Chrome and Firefox together in three runs on the same evening. In a Linux container sharing a network with CloakBrowser, Firefox took 5 to 12 seconds to run the smoke test, and 6 to 13 without it. The browsers job starts Chrome and Firefox on the same image, so each browser is still started on every pull request. What this repository's CI no longer does is start Chrome and Firefox beside the container.

Verification

Build, with dotnet build -c Release.

Build succeeded.
    0 Warning(s)
    0 Error(s)

Unit set, as this repository's CI runs it, with no variables set. main gives 39 passed and 15 skipped, so the 18 new results are the new unit tests and the skips are unchanged.

dotnet test -c Release --no-build --filter "TestCategory!=Contract&TestCategory!=CloudInternal&TestCategory!=Browser"
Passed!  - Failed:     0, Passed:    57, Skipped:    15, Total:    72

Cloak tests of the Browser category on Windows, attached to the container through Docker Desktop.

  Passed Cloak_RunsThePage [2 s]
   Browser reported: ran in Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36
  Passed Cloak_StartsEachTestClean [3 s]
Total tests: 2
     Passed: 2

Cloak_StartsEachTestClean fails when the reset is taken out of CreateCloak, which is what makes the clean start a checked claim.

  Failed Cloak_StartsEachTestClean [2 s]
   Assert.AreEqual failed. Expected string length 29 but was 46. ... the browser kept what the driver before this one left

The same test on Linux, with the vendor container and the suite sharing a host network, no Chrome on the machine and SE_AVOID_BROWSER_DOWNLOAD=true.

  Passed Cloak_RunsThePage [825 ms]
   Browser reported: ran in Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36

With the variable unset every Cloak test is skipped and says why.

  Skipped Cloak_RunsThePage, because: No CloakBrowser to attach to. Set CLOAK_DEBUGGER_ADDRESS to the host and port of one that is running, for example 127.0.0.1:9222. The README says how to start one.
  Skipped SessionStorageCache_Cloak (True), because: No CloakBrowser to attach to. ...
  Skipped JavaScript_Get_ValidLicense, because: No CloakBrowser to attach to. ...

With the variable set to an address where nothing listens, the test fails and names the variable and the address.

Contract category against the .NET cloud example on Windows, with CLOAK_DEBUGGER_ADDRESS set. The user agent arguments in the test names are trimmed here.

  Passed Example_RendersClientSideOverridesOnThePage (375,667,2,"...") [7 s]
  Passed Example_RendersClientSideOverridesOnThePage (414,736,3,"...") [5 s]
  Passed Example_RendersClientSideOverridesOnThePage_Cloak [9 s]
  Passed Example_ServesCoreJs_AndClientSideOverridesFlow (375,667,2,"...") [3 s]
  Passed Example_ServesCoreJs_AndClientSideOverridesFlow (414,736,3,"...") [3 s]
  Passed Example_ServesCoreJs_AndClientSideOverridesFlow_Cloak [2 s]
  Passed Example_RendersRealDetectionResult [2 s]
  Passed Example_RendersRealDetectionResult_Cloak [1 s]
  Passed SessionStorageCache_Chrome (True) [3 s]
  Passed SessionStorageCache_Chrome (False) [4 s]
  Passed SessionStorageCache_FireFox (True) [11 s]
  Passed SessionStorageCache_FireFox (False) [11 s]
  Passed SessionStorageCache_Cloak (True) [3 s]
  Passed SessionStorageCache_Cloak (False) [3 s]
Total tests: 14
     Passed: 14

The same command with the variable unset, as the control. The nine Chrome and Firefox results pass as above.

  Skipped Example_RendersClientSideOverridesOnThePage_Cloak
  Skipped Example_ServesCoreJs_AndClientSideOverridesFlow_Cloak
  Skipped Example_RendersRealDetectionResult_Cloak
  Skipped SessionStorageCache_Cloak (True)
  Skipped SessionStorageCache_Cloak (False)
Total tests: 14
     Passed: 9
    Skipped: 5

The CloakBrowser results with the script's call back refused, which is finding 10.

  Failed Example_RendersClientSideOverridesOnThePage_Cloak [8 s]
   ... the screen width rendered on the page does not match the width of the browser's screen
Expected: "1920"
But was:  "0"
  Failed Example_ServesCoreJs_AndClientSideOverridesFlow_Cloak [2 s]
   ... screenpixelswidth override did not resolve
  Passed Example_RendersRealDetectionResult_Cloak [2 s]
  Failed SessionStorageCache_Cloak (True) [3 s]
   ... the device id rendered on the second page must come from the cached results
  Failed SessionStorageCache_Cloak (False) [3 s]
   ... no json refresh call is expected on the second page
Total tests: 5
     Passed: 1
     Failed: 4

Example_RendersClientSideOverridesOnThePage_Cloak with the width read from the browser forced to 0.

  Failed Example_RendersClientSideOverridesOnThePage_Cloak [1 s]
   Assert.IsTrue failed. ... the browser reports a screen of 0 by 1080, so the client-side results cannot be told from the server's

The cloak job ran on a real ubuntu-latest runner (image ubuntu-24.04, version 20260927.320.1) and passed on the latest commit, in this run. It pulled the image by digest, and its log shows these lines.

Digest: sha256:2fdd1289154f30594c7bab74943ea7baa6b617a9ec026ab4bf339c2ebabcb712
{"Browser": "Chrome/146.0.7680.177", "Protocol-Version": "1.3", ...
  Passed Cloak_RunsThePage [2 s]
  Passed Cloak_StartsEachTestClean [804 ms]
Total tests: 2
     Passed: 2

Two earlier runs did not pass first time. In the first, the three ubuntu-latest jobs were cancelled before any step ran, with "The job was not acquired by Runner of type hosted even after multiple attempts", while GitHub was reporting an incident with Actions, and they passed when re-run. In the second, Firefox took more than 60 seconds to start in the cloak job, which is why that job no longer starts Firefox.

What was not run

• CloakTests was not run in CloakBrowser, on purpose. The script it runs in the browser carries the resource key and the license (finding 12), and there were no keys at hand that are meant to be seen by a third party's browser. The class compiles, is discovered, and is skipped when the variable is unset. Its first run belongs to a pipeline that supplies such keys.
• Only the .NET cloud example was run. The examples of the other five languages, and the on-premise examples, have not been loaded in CloakBrowser.
• The Contract runs were made on Windows with Docker Desktop. On Linux with host networking only the Browser category has run, in this repository's CI.
• The resource key used does not carry every property the example displays. It carries the ones the Contract tests assert on, which was confirmed with the cloud's accessible properties call before the run, and the example logged five others as not available with that key. No test reads those rows.
• Docker Desktop on macOS and the linux/arm64 build of the image were not tried.

What remains

• Each repository that runs this suite has to start the container and set CLOAK_DEBUGGER_ADDRESS in its own CI before its examples are tested in CloakBrowser. That is follow-up work in those repositories and is not part of this pull request.
• Before a repository runs Contract under CloakBrowser, it is worth checking that its example gives the browser no key, as finding 11 did for the .NET cloud example.
• A pipeline that runs CloudInternal under CloakBrowser needs keys it is content for a third party's browser to see.
• Firefox's start on the ubuntu-latest image comes close to Selenium's limit of 60 seconds at times, with no CloakBrowser involved. The browsers job took 46, 48 and 57 seconds for Chrome and Firefox together in three runs here, against 18 to 34 seconds in four runs in September. That is a risk to the browsers job and to every Contract run that uses Firefox, and it is not addressed here.
• Host networking opens the DevTools port on every interface of the machine. That is acceptable on a hosted CI runner and is called out in the README for anywhere else.
• When the image pin moves, the license link in the README has to be checked against the file in the new image.
• The run carries a notice from GitHub that the ubuntu-latest label moves to Ubuntu 26 from 19 October 2026 (runner-images issue 14748), so the cloak job is worth watching on its first run after that.