[jehrr]

Problem

Every measured refusal on Vrbo is Expedia's "Bot or Not?" handler picking DataDome. Until now the repo only reported that as blocked (exit 3), and its single 2026-09-14 experiment had stopped at a bare cookie pasted behind the page.

Resulting behaviour

When the handler picks DataDome, all three engines go through these steps:

1. Settle. The page gets up to 15 s to clear by itself (its own DataDome tag can pass without a slider) or to mount the slider. A page that clears costs nothing.
2. Check before buying. The slider's iframe URL is read off the live page. Nothing is bought when any of these holds:
   • t=bv, which 2Captcha's docs say means the exit is banned;
   • no TWOCAPTCHA_KEY;
   • no proxy the browser itself exits through: over --cdp-endpoint, with no proxy, or Selenium with a proxy login (Selenium strips the login, so that proxy is not its exit);
   • a page reached by a button press;
   • the page's one-solve allowance is already spent.
3. Solve. It sends a DataDomeSliderTask with websiteURL, captchaUrl, the browser's own navigator.userAgent and the browser's current proxy exit (the sticky -session- login from Keep one proxy exit for the whole run on a 2Captcha gateway #20 included). The fields follow 2Captcha's current DataDome docs, checked 2026-10-23. The task is polled for at most 180 s. The allowance is claimed before createTask and never given back.
4. Through the widget. The answer is reported the way the slider reports its own pass: a trusted postMessage from inside the slider's captcha-delivery iframe, {"eventType":"passed","responseType":"captcha","cookie":…}.
   • The site's own DataDome tag (tags.js 5.11.1) then installs the cookie and its x-datadome-clientid session, and fires dd_response_passed.
   • On that event the handler page makes its own POST /botOrNot/validate and leaves for the search. The engine gives this 20 s.
   • Playwright and Selenium can post from the iframe. pyppeteer cannot reach a cross-origin iframe, so it skips this step.
5. Cookie and re-load, then validate. If the page is still the challenge, the engine installs the cookie with the challenge page's own cookie domain, path, Secure and SameSite, reads it back, and re-loads with the same settle. If the handler page is still there, the engine makes the validate call itself (X-Exchange-Handshake plus the page config's destination / exchangeHandshake).

The widget message and the validate call are reproduced from the site's served tag and challenge bundle, read 2026-10-23. Neither was taken from a captured exchange. responseType: "captcha" is an inference.
6. Record. The sidecar gains these fields:

• captcha_kind;
• captcha_solves (tasks created);
• captcha_solved (answers returned);
• captcha_accepted (answers after which the page classified as served);
• captcha_attempts (one record per decision, with the outcome, cookie applied/kept, re-load and validate statuses, and accepted).

<out>.latest_attempt.meta.json is now written for any run that met a captcha.

A DataDome page now classifies as challenge instead of blocked. An uncleared one still exits 3, with blocked_by: datadome-challenge.

The canary's pagination job now receives TWOCAPTCHA_KEY, passes --solve-captcha when-blocked explicitly, adds the captcha fields to the step summary and uploads the attempt sidecars. It warns, without failing, when the key is absent. The first-page job gets no key and no proxy, so its records are the free no-solve control.

README, TROUBLESHOOTING, CONTRIBUTING, .env.example, --help and the CHANGELOG under [Unreleased] now describe this path and its limits. The README keeps the 2026-09-14 control table as history.

Evidence

Offline suite (python smoke_test.py), with no request leaving 127.0.0.1 and no key sent anywhere:

• New groups:
  • verdicts on the slider URL;
  • task fields against the docs;
  • the solver API stubbed on 127.0.0.1: polling, error codes, bounded timeout, no key or proxy password in any message;
  • cookie attribute mirroring;
  • the validate request read from the real sanitized handler fixture;
  • the fallback through a scripted browser: the widget message and widget-cleared, widget-did-nothing then the cookie path, accepted, refused-then-validate, refused, self-cleared, t=bv, no key, no proxy, no address, four attempts on one budget = 1 purchase;
  • the sidecar;
  • each engine's _solver_proxy;
  • each engine's real _fetch_one_page through retries and rotation: exactly 1 purchase with Playwright and pyppeteer, 0 with Selenium plus a login proxy, 0 over --cdp-endpoint, accepted recorded only on a served re-load.
• It passed in a core-only venv and in a Playwright venv locally. It also passed in a pyppeteer venv and in a Selenium venv, each with the sibling engines skipped.
• Opt-in real-browser group (DATADOME_BROWSER=1, local Chromium 154). It runs against a stub site and a stub API on 127.0.0.1, with the slider frame served over local HTTPS and a stub of the tag's trusted-message handler. It covers three cases (accepted through the widget, refused everywhere, the cookie path alone), and all checks passed for each of Playwright, pyppeteer and Selenium:
  • the captchaUrl is read off the live iframe;
  • the solver gets the exact User-Agent the site received;
  • the pass is posted from inside the frame, and the page's handler validates with the solved cookie (Playwright and Selenium; pyppeteer is recorded as widget_posted: false and falls through);
  • the cookie is set and read back with SameSite=Lax and path /;
  • our validate POST carries the handshake header and the solved cookie;
  • each result is recorded as accepted or as solved-but-refused.
• 10 planted-fault controls. Each made the suite red on its named check, and the restored tree was green:
  1. the budget is not consulted;
  2. t=bv is bought;
  3. the UA is not passed;
  4. the page cookie attributes are ignored;
  5. validate is never called;
  6. Playwright hands no proxy to the solver;
  7. Playwright never settles acceptance;
  8. Selenium hands over its stripped login proxy;
  9. a solve is bought with no proxy;
  10. the widget pass message is misnamed.
• ci_checks.py --all passed, including the secret scan.

Live verification (paid, bounded)

These were 7 canary dispatches on this branch, 2026-10-23. Run 1 was on dcc7eff, before the widget step existed. Runs 2-7 were on 19f5631. Each walk used the VRBO_PROXY residential exit pinned to one sticky session, headful real Chrome, --block-media and a 15 MB budget. At most one solve per page was bought.

run	DataDome on walk p1	solve created / returned	widget step	cookie applied → re-load → our validate	page accepted	pages	proxy	billed
39925584563	yes	1 / 1 (26.8 s)	not in that build	applied → 429, cookie replaced → 403	no	0	3.4 MB	$0.00145
39927514698	no (served, grid never painted, exit 5)	0	—	—	—	0	7.0 MB	0
39928080711	yes	1 / 1 (37.1 s)	page's own validate 200, left the challenge	not needed	yes	3 (150 rows, complete)	10.0 MB	$0.00145
39928701642	yes	1 / 1 (31.5 s)	page's validate 403	applied → 429, replaced → 403	no	0	3.4 MB	$0.00145
39929182026	yes	1 / 1 (26.8 s)	page's validate 403	applied → 429, replaced → 403	no	0	3.7 MB	$0.00145
39929627110	yes	1 / 1 (26.6 s)	page's validate 403	applied → 429, replaced → 403	no	0	4.2 MB	$0.00145
39931458527	yes	1 / 1 (42.5 s)	page's own validate 200, left the challenge	not needed	yes	3 (150 rows, complete)	9.3 MB	$0.00145

The totals:

• 7 runs; DataDome was served on the walk's page 1 in 6.
• 6 solves were billed ($0.0087).
• 2 were accepted, and those two walks obtained 6 pages.
• Proxy traffic was 41.0 MB, every run under its 15 MB cap.

No-solve control: the first-page job, from the runner's own address with no key and no proxy, met DataDome on all 7 runs. It never cleared by itself in either of its two tries per run (no_key / no_widget records).

Conclusion: the fallback works, but not reliably.

• Two of five answers reported through the widget were accepted. Both times the page's own validate call answered 200 right after the post, and the walk completed. Before this, the walk had never completed in this repo's canary.
• Pasting the cookie and re-loading, plus our own validate call, was accepted 0 times out of 4. The site replaced the cookie and mounted a new slider every time.
• The refusals come at the handler's validate (HTTP 403). Nothing recorded shows why one answer is accepted and another refused.
• Seven runs are too few for a rate.

I stopped at 7 of the 10 allowed dispatches, because the pattern was already clear: accepted only through the widget, refused at the validate step otherwise.

Not changed

• No release, version bump or tag.
• The browserless clients (scraper_api_client.py) get no DataDome path: there is no browser to carry the cookie.
• Selenium with a proxy login cannot solve. That is a limit of Selenium's proxy handling, described in --help and the README.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8