From 0931f6c80a209875b98ef6c85d638a61bcedd3bc Mon Sep 17 00:00:00 2001 From: Petr Date: Thu, 22 Oct 2026 22:08:21 +0000 Subject: [PATCH] CDP: leave a connected browser running, never hang connecting, retry a 500 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two defects from a manual test of v0.2.0, both reproduced on a local Chrome started with --remote-debugging-port before fixing. pyppeteer shut down the browser it connected to. Its teardown called `Browser.close()`, which for a connected browser sends CDP `Browser.close`: the browser on the other end died, `Event loop is closed` tracebacks were printed, and the next run hung. Reproduced: browser DEAD after run 1, run 2 killed by a 60 s timeout. A connected browser now has its page closed and the connection dropped (`disconnect()`); only a browser the engine launched is closed. The connection's tasks are drained before the event loop is closed. It also took the remote browser's own first tab (`pages()[0]`) and navigated it away; it now opens a new page. The hang itself was pyppeteer's `connect()`: it opens the WebSocket in a background task and waits for the answer to its first command, so a refused port or an HTTP 500 handshake died in that task and `connect()` waited forever. Measured against a local server answering 500 and against a closed port: both hung. The engine now connects through a small wrapper that waits on the command AND the background task and raises the real reason — `HTTP 500`, `Connection refused`, or a handshake that never completed — within the timeout. Playwright left a tab per run in a browser it connected to: `Browser.close()` over CDP only disconnects. It now closes the page it opened, and the context if it created one. Reproduced: 1 -> 2 -> 3 tabs over two runs; after the fix 1 -> 1 -> 1. Both engines now retry a CDP handshake refused with HTTP 500, 502 or 503, twice, after 3 s and 6 s — a managed browser can refuse while a previous session on the same profile closes. The policy is data in `page_flow`; 401, a closed port and a timeout fail at once. Verified: - local Chrome, two runs per engine: browser ALIVE, tabs 1 -> 1, 32 rows each, no tracebacks. - local server answering 500: both engines retry twice and exit 5 in 9 s with the status in the message; closed port: exit 5 at once; an endpoint that accepts TCP and never answers: pyppeteer exits 5 in 20 s instead of hanging. A password planted in the endpoint never reached the output. - NOT verified live on the Scraping Browser API: every endpoint in the workspace answered 401 (expired credentials), which both engines now report at once without retrying. - non-CDP runs of both engines unchanged. - tests: the retry policy as data; each driver's teardown on recorded fakes; pyppeteer's connect against a real local 500 server and a closed port; the retry loops. Six controls in the engine venvs, each caught by its named check; two tests were changed to report a raised error by name after a control showed them crashing instead. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01VrpkXSoXyZMGV5nqF9JV6P --- CHANGELOG.md | 19 ++++ README.md | 2 +- page_flow.py | 48 ++++++++++ playwright_scraper.py | 68 ++++++++++++- puppeteer_scraper.py | 146 ++++++++++++++++++++++++++-- smoke_test.py | 216 ++++++++++++++++++++++++++++++++++++++++++ 6 files changed, 484 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c1f852a..c897a6d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -79,6 +79,25 @@ discovered from a bill or a diff. matches (`git check-ignore --no-index`). Without that it never reported a file already tracked, so it only worked until the fixture was committed. +- **pyppeteer no longer shuts down a browser it connected to.** Over + `--cdp-endpoint` its teardown sent CDP `Browser.close`, killing the browser + on the other end — a local Chrome, or the session on the Scraping Browser + API — and printing `Event loop is closed` tracebacks; the next run then + hung. A connected browser now has its page closed and the connection + dropped. It also took the remote browser's own first tab and navigated it + away; it now opens a new page. +- **A CDP endpoint that refuses the handshake no longer hangs pyppeteer.** + pyppeteer's own `connect()` swallowed a refused port or an HTTP 500 into a + background task and waited for an answer that never came. The connection + now raises the real reason (`HTTP 500`, `Connection refused`, or a + handshake that never completed) within the timeout. +- **Playwright closes the page it opened over CDP.** One tab was left in the + remote browser per run (1 → 2 → 3 over two runs on a local Chrome). +- **Both engines retry a CDP handshake refused with HTTP 500, 502 or 503** + twice, after 3 s and 6 s. A managed browser can refuse for a few seconds + while a previous session on the same profile closes. Other refusals — + 401, a closed port, a timeout — fail at once. + ### Changed - CI only: the canary now runs weekly (Mondays 06:17 UTC) instead of daily, and on a manual dispatch. diff --git a/README.md b/README.md index ccf5a7c..8099eec 100644 --- a/README.md +++ b/README.md @@ -207,7 +207,7 @@ package ids were identical. | **Playwright** (primary) | `requirements-playwright.txt` | the one to use | | Selenium | `requirements-selenium.txt` | no authenticated remote CDP endpoint (`--cdp-endpoint` is refused, exit 2); `--proxy-server` cannot authenticate at all, so credentials are stripped and a warning printed; exposes no HTTP status, so classification leans on the structural signals (Play's own 404 template is recognised from its body); `--fingerprint` is refused | | pyppeteer | `requirements-puppeteer.txt` | effectively unmaintained upstream; here for parity; `--fingerprint` is refused | -| Scraping Browser API | `--cdp-endpoint` | a managed browser; never stack `--proxy` or `--fingerprint` on it | +| Scraping Browser API | `--cdp-endpoint` | a managed browser; never stack `--proxy` or `--fingerprint` on it. Playwright and pyppeteer open their own page on it and leave the browser running when the run ends; a handshake refused with HTTP 500/502/503 is retried twice (3 s, 6 s), anything else fails at once with exit 5 | **Install exactly one per virtualenv.** Playwright and pyppeteer declare mutually unsatisfiable pins (`pyee` <12 vs ≥13) and pyppeteer and selenium diff --git a/page_flow.py b/page_flow.py index d93a085..13af004 100644 --- a/page_flow.py +++ b/page_flow.py @@ -17,6 +17,7 @@ primitives in and this module names the OPERATION. """ +import re from typing import Any, Callable, Dict, Iterable, List, Optional, Tuple import urllib.parse @@ -347,3 +348,50 @@ def block_advice(html: Optional[str], headless: bool, "here — but if a reCAPTCHA does appear on a refused page, the " "run will solve it with a key (at most one paid solve per page)") return tips + + +# -------------------------------------------------------------------------- +# Connecting to a browser over CDP +# -------------------------------------------------------------------------- + +# A managed browser can refuse the WebSocket handshake for a few seconds +# after a previous session on the same profile ended: the 2Captcha Scraping +# Browser API answers HTTP 500 while it tears the old session down (reported +# by a manual test of 0.2.0, back-to-back runs). That refusal costs nothing — +# no session was created — and clears on its own, so it is worth a short wait +# and another try. A refused port or a timeout is not: the first says nothing +# is there, the second has already waited the full timeout. +CDP_CONNECT_ATTEMPTS = 3 +CDP_CONNECT_BACKOFF_S = (3.0, 6.0) +_CDP_RETRIABLE_STATUSES = (500, 502, 503) + +_STATUS_IN_TEXT = re.compile(r"\b(50[023])\b") + + +def cdp_connect_status(exc: BaseException) -> Optional[int]: + """The HTTP status a failed CDP handshake carried, if it says. + + `websockets` reports it as `status_code` (10.x) or `response.status_code` + (newer); Playwright only in its message — "WebSocket error: 500 + Internal Server Error", measured against a local server answering 500. + """ + for value in (getattr(exc, "status_code", None), + getattr(getattr(exc, "response", None), "status_code", None)): + if isinstance(value, int): + return value + text = str(exc) + if "WebSocket" in text or "HTTP" in text or "handshake" in text.lower(): + m = _STATUS_IN_TEXT.search(text) + if m: + return int(m.group(1)) + return None + + +def cdp_connect_retriable(exc: BaseException) -> bool: + return cdp_connect_status(exc) in _CDP_RETRIABLE_STATUSES + + +def cdp_connect_delay(attempt: int) -> float: + """Seconds to wait before attempt `attempt + 1` (attempt counts from 1).""" + idx = max(0, min(attempt - 1, len(CDP_CONNECT_BACKOFF_S) - 1)) + return CDP_CONNECT_BACKOFF_S[idx] diff --git a/playwright_scraper.py b/playwright_scraper.py index 6e8593d..133c588 100644 --- a/playwright_scraper.py +++ b/playwright_scraper.py @@ -26,6 +26,7 @@ """ import sys +import time from typing import Any, Dict, Optional, Tuple # Imported at MODULE level on purpose: the smoke suite must be able to skip @@ -46,11 +47,17 @@ class PlaywrightDriver: name = NAME - def __init__(self, pw_ctx, browser, context, page) -> None: + def __init__(self, pw_ctx, browser, context, page, connected: bool = False, + owns_context: bool = True) -> None: self._pw_ctx = pw_ctx self._browser = browser self._context = context self._page = page + # Whether the browser was reached over --cdp-endpoint rather than + # launched here, and so which parts of it this run may tear down. + self._connected = connected + self._owns_context = owns_context + self._owns_page = True def fetch(self, url: str, mode: str) -> Tuple[str, Optional[int]]: """One navigation. @@ -95,12 +102,56 @@ def current_url(self) -> str: return self._page.url def close(self) -> None: + """Leave the browser the way it was found. + + A browser this engine LAUNCHED is closed. One it CONNECTED to over + `--cdp-endpoint` belongs to somebody else: only the page — and the + context, if this run created it — are closed, and then the connection + is dropped. `Browser.close()` on a CDP connection only disconnects, so + the page opened for the run stayed open in the remote browser: one + more tab per run (1 -> 2 -> 3 over two runs, measured 2026-10-22 on a + local Chrome with --remote-debugging-port). + """ try: + if self._connected: + for owned in (self._page if self._owns_page else None, + self._context if self._owns_context else None): + if owned is None: + continue + try: + owned.close() + except PlaywrightError: + pass # already gone with the remote session self._browser.close() finally: self._pw_ctx.__exit__(None, None, None) +def _connect_over_cdp(pw, endpoint: str): + """`connect_over_cdp`, retrying the handshake refusals worth retrying. + + Playwright reports a refused handshake at once ("WebSocket error: ... + 500 Internal Server Error"), so the policy in `page_flow` decides whether + to wait and try again; anything else is raised to the caller as it was. + """ + attempt = 0 + while True: + attempt += 1 + try: + return pw.chromium.connect_over_cdp( + endpoint, timeout=engine_core.DEFAULT_TIMEOUT_MS) + except PlaywrightError as exc: + if (attempt >= page_flow.CDP_CONNECT_ATTEMPTS + or not page_flow.cdp_connect_retriable(exc)): + raise + delay = page_flow.cdp_connect_delay(attempt) + print(f"[warn] the CDP endpoint refused the connection (HTTP " + f"{page_flow.cdp_connect_status(exc)}); retrying in " + f"{delay:.0f}s ({attempt}/{page_flow.CDP_CONNECT_ATTEMPTS})", + file=sys.stderr) + time.sleep(delay) + + def launch(args, proxy_url: Optional[str]) -> PlaywrightDriver: """A browser, a context and a page. @@ -115,10 +166,17 @@ def launch(args, proxy_url: Optional[str]) -> PlaywrightDriver: # Never set a UA, a fingerprint or a proxy over CDP: the remote # browser brings its own identity and stacking a second one # manufactures a contradiction rather than better cover. - browser = pw.chromium.connect_over_cdp( - args.cdp_endpoint, timeout=engine_core.DEFAULT_TIMEOUT_MS) - context = browser.contexts[0] if browser.contexts else browser.new_context() - return PlaywrightDriver(pw_ctx, browser, context, context.new_page()) + try: + browser = _connect_over_cdp(pw, args.cdp_endpoint) + except BaseException: + pw_ctx.__exit__(None, None, None) + raise + owns_context = not browser.contexts + context = browser.new_context() if owns_context else browser.contexts[0] + # A NEW page, always: the remote browser's existing tabs are its + # owner's, and this run closes only what it opened. + return PlaywrightDriver(pw_ctx, browser, context, context.new_page(), + connected=True, owns_context=owns_context) context_kwargs: Dict[str, Any] = {} init_script = None diff --git a/puppeteer_scraper.py b/puppeteer_scraper.py index 0ee0dac..7392e27 100644 --- a/puppeteer_scraper.py +++ b/puppeteer_scraper.py @@ -51,10 +51,13 @@ class PuppeteerDriver: name = NAME - def __init__(self, loop, browser, page) -> None: + def __init__(self, loop, browser, page, connected: bool = False) -> None: self._loop = loop self._browser = browser self._page = page + # Reached over --cdp-endpoint rather than launched here: the browser + # is somebody else's, and only the page this run opened is ours. + self._connected = connected def _run(self, coro): return self._loop.run_until_complete(coro) @@ -114,10 +117,132 @@ def current_url(self) -> str: return self._page.url or "" def close(self) -> None: + """Leave the browser the way it was found. + + For a CONNECTED browser pyppeteer's `Browser.close()` sends CDP + `Browser.close` — it shut down the browser on the other end of + `--cdp-endpoint`, whoever's it was, and the next run then hung on a + dead endpoint (measured 2026-10-22 on a local Chrome with + --remote-debugging-port; reported on the Scraping Browser too). A + connected browser now gets its page closed and the connection dropped; + only a browser this engine launched is closed. + + The connection's own tasks are then let finish before the event loop + is closed. Closing the loop under them printed `Event loop is closed` + tracebacks after every CDP run. + """ try: - self._run(self._browser.close()) + if self._connected: + try: + self._run(self._page.close()) + except Exception: # the session may already be gone + pass + self._run(self._browser.disconnect()) + else: + self._run(self._browser.close()) finally: - self._loop.close() + _drain_and_close(self._loop) + + +class CDPConnectError(ConnectionError): + """The CDP endpoint could not be connected to, with the reason it gave.""" + + def __init__(self, message: str, status_code: Optional[int] = None) -> None: + super().__init__(message) + self.status_code = status_code + + +async def _connect(endpoint: str, timeout_s: float): + """pyppeteer's `connect()`, but a refused handshake RAISES. + + pyppeteer opens the WebSocket in a background task and then waits for the + answer to its first CDP command. When the handshake fails — a refused + port, an HTTP 500 from a managed browser — that background task dies with + the reason and the command is never answered, so `connect()` waited + forever: the "next run hangs" in the report. Measured 2026-10-22 against + a local server answering 500 and against a closed port: both hung, with + the real error visible only in a "Task exception was never retrieved" + traceback at exit. + + This waits on the first command AND on the background task, and raises + whichever finishes first with an error, bounded by `timeout_s`. It relies + on `Connection._recv_fut`, a private attribute present in every pyppeteer + release this repository supports; without it, it degrades to the timeout. + """ + from pyppeteer.browser import Browser + from pyppeteer.connection import Connection + + loop = asyncio.get_event_loop() + connection = Connection(endpoint, loop, 0) + # `send()` returns a Future, not a coroutine, so `ensure_future` and not + # `create_task`. + first = asyncio.ensure_future(connection.send("Target.getBrowserContexts")) + receiver = getattr(connection, "_recv_fut", None) + waiting = {first} | ({receiver} if receiver is not None else set()) + done, _pending = await asyncio.wait(waiting, timeout=timeout_s, + return_when=asyncio.FIRST_COMPLETED) + if first not in done: + reason = None + if receiver is not None and receiver in done and not receiver.cancelled(): + reason = receiver.exception() + # Dispose BEFORE touching `first`: it is the very future pyppeteer + # keeps for the pending command, and `dispose()` fails it with + # "Target closed" — cancelling it first made that an InvalidStateError + # that replaced the real reason. + try: + await connection.dispose() + except Exception: + pass + if first.done() and not first.cancelled(): + first.exception() # retrieved, so it is not reported at exit + else: + first.cancel() + if isinstance(reason, (asyncio.TimeoutError, TimeoutError)): + raise CDPConnectError( + "the CDP endpoint accepted the connection and never completed " + "the WebSocket handshake") from None + if reason is not None: + raise CDPConnectError( + f"the CDP endpoint refused the connection: " + f"{type(reason).__name__}: {reason}", + page_flow.cdp_connect_status(reason)) from None + raise CDPConnectError( + f"no answer from the CDP endpoint within {timeout_s:.0f}s") + contexts = first.result().get("browserContextIds", []) + return await Browser.create(connection, contexts, False, + {"width": 800, "height": 600}, None, + lambda: connection.send("Browser.close")) + + +def _connect_with_retry(loop, endpoint: str): + timeout_s = engine_core.DEFAULT_TIMEOUT_MS / 1000.0 + attempt = 0 + while True: + attempt += 1 + try: + return loop.run_until_complete(_connect(endpoint, timeout_s)) + except CDPConnectError as exc: + if (attempt >= page_flow.CDP_CONNECT_ATTEMPTS + or not page_flow.cdp_connect_retriable(exc)): + raise + delay = page_flow.cdp_connect_delay(attempt) + print(f"[warn] the CDP endpoint refused the connection (HTTP " + f"{exc.status_code}); retrying in {delay:.0f}s " + f"({attempt}/{page_flow.CDP_CONNECT_ATTEMPTS})", file=sys.stderr) + loop.run_until_complete(asyncio.sleep(delay)) + + +def _drain_and_close(loop) -> None: + """Let the connection's own tasks finish, then close the loop.""" + try: + pending = [t for t in asyncio.all_tasks(loop) if not t.done()] + for task in pending: + task.cancel() + if pending: + loop.run_until_complete( + asyncio.gather(*pending, return_exceptions=True)) + finally: + loop.close() def launch(args, proxy_url: Optional[str]) -> PuppeteerDriver: @@ -125,12 +250,15 @@ def launch(args, proxy_url: Optional[str]) -> PuppeteerDriver: asyncio.set_event_loop(loop) if args.cdp_endpoint: - from pyppeteer import connect - browser = loop.run_until_complete( - connect(browserWSEndpoint=args.cdp_endpoint)) - pages = loop.run_until_complete(browser.pages()) - page = pages[0] if pages else loop.run_until_complete(browser.newPage()) - return PuppeteerDriver(loop, browser, page) + try: + browser = _connect_with_retry(loop, args.cdp_endpoint) + # A NEW page, always. This used to take `pages()[0]` — the + # remote browser's own first tab — and navigate it away. + page = loop.run_until_complete(browser.newPage()) + except BaseException: + _drain_and_close(loop) + raise + return PuppeteerDriver(loop, browser, page, connected=True) launch_args = ["--disable-blink-features=AutomationControlled", "--window-size=1440,2400"] diff --git a/smoke_test.py b/smoke_test.py index a67295d..faa4497 100644 --- a/smoke_test.py +++ b/smoke_test.py @@ -1771,6 +1771,219 @@ def make_blocker(tmp): return ok +def test_cdp_connect_policy(): + """Which CDP handshake refusals are worth waiting out, as data.""" + ok = True + + class WithStatus(Exception): + status_code = 500 + + class Response: + status_code = 503 + + class WithResponse(Exception): + response = Response() + + cases = ( + ("Playwright's 500 message", + Exception("BrowserType.connect_over_cdp: WebSocket error: " + "ws://h:9222/ 500 Internal Server Error"), True), + ("websockets status_code 500", WithStatus("rejected"), True), + ("websockets response.status_code 503", WithResponse("rejected"), True), + ("401 Unauthorized", Exception("WebSocket error: ws://h:9222/ 401 " + "Unauthorized"), False), + ("a refused port", ConnectionRefusedError(111, "Connect call failed"), False), + ("a timeout", TimeoutError(), False), + ("a 500 in unrelated text", Exception("page 500 of results"), False), + ) + for label, exc, want in cases: + ok &= check(f"[cdp] retriable({label}) is {want}", + page_flow.cdp_connect_retriable(exc) is want) + ok &= check("[cdp] bounded attempts", page_flow.CDP_CONNECT_ATTEMPTS == 3) + return ok + + +class _Recorder: + """A fake browser object: every method call is recorded, sync or async.""" + + def __init__(self, log, name, is_async=False): + self._log, self._name, self._async = log, name, is_async + + def __getattr__(self, attr): + def call(*a, **k): + self._log.append(f"{self._name}.{attr}") + if self._async: + async def done(): + return None + return done() + return None + return call + + +def test_cdp_teardown_playwright(): + """Over --cdp-endpoint, close only what this run opened.""" + ok = True + try: + import playwright_scraper as engine + except ImportError as exc: + skip("cdp-playwright", f"playwright not installed ({exc})") + return True + + class Ctx: + def __exit__(self, *a): + log.append("playwright.stop") + + for connected, owns_context, want in ( + (True, False, ["page.close", "browser.close", "playwright.stop"]), + (True, True, ["page.close", "context.close", "browser.close", + "playwright.stop"]), + (False, True, ["browser.close", "playwright.stop"])): + log = [] + driver = engine.PlaywrightDriver( + Ctx(), _Recorder(log, "browser"), _Recorder(log, "context"), + _Recorder(log, "page"), connected=connected, + owns_context=owns_context) + driver.close() + ok &= check(f"[cdp] playwright close (connected={connected}, " + f"owns_context={owns_context})", log == want, str(log)) + + # The handshake retry: two 500s, then a browser. + calls = [] + + class Chromium: + def connect_over_cdp(self, endpoint, timeout): + calls.append(endpoint) + if len(calls) < 3: + raise engine.PlaywrightError( + "BrowserType.connect_over_cdp: WebSocket error: " + "ws://h:9222/ 500 Internal Server Error") + return "BROWSER" + + class PW: + chromium = Chromium() + + saved = page_flow.CDP_CONNECT_BACKOFF_S + page_flow.CDP_CONNECT_BACKOFF_S = (0.0, 0.0) + try: + try: + with redirect_stderr(io.StringIO()): + got = engine._connect_over_cdp(PW(), "ws://h:9222/") + except Exception as exc: # report it by name, not as a traceback + got = f"raised {type(exc).__name__}" + ok &= check("[cdp] playwright retries a 500 handshake, then connects", + got == "BROWSER" and len(calls) == 3, f"{got} {len(calls)}") + calls.clear() + + class Chromium401(Chromium): + def connect_over_cdp(self, endpoint, timeout): + calls.append(endpoint) + raise engine.PlaywrightError("WebSocket error: ws://h/ 401 Unauthorized") + + PW.chromium = Chromium401() + try: + engine._connect_over_cdp(PW(), "ws://h:9222/") + ok &= check("[cdp] playwright raises a 401", False, "no error") + except engine.PlaywrightError: + ok &= check("[cdp] playwright does not retry a 401", len(calls) == 1, + str(len(calls))) + finally: + page_flow.CDP_CONNECT_BACKOFF_S = saved + return ok + + +def test_cdp_teardown_pyppeteer(): + """Disconnect from a browser this run did not launch; never hang connecting.""" + ok = True + try: + import puppeteer_scraper as engine + except ImportError as exc: + skip("cdp-pyppeteer", f"pyppeteer not installed ({exc})") + return True + import asyncio + import http.server + import socket + import threading + import time + + for connected, want in ((True, ["page.close", "browser.disconnect"]), + (False, ["browser.close"])): + log = [] + loop = asyncio.new_event_loop() + driver = engine.PuppeteerDriver(loop, _Recorder(log, "browser", True), + _Recorder(log, "page", True), + connected=connected) + driver.close() + ok &= check(f"[cdp] pyppeteer close (connected={connected})", + log == want, str(log)) + ok &= check(f"[cdp] pyppeteer closes its loop (connected={connected})", + loop.is_closed()) + + # A real handshake refusal must RAISE, with its status, not hang. + class Refuse(http.server.BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + + def do_GET(self): + self.send_response(500) + self.send_header("Content-Length", "0") + self.end_headers() + + def log_message(self, *a): + pass + + server = http.server.HTTPServer(("127.0.0.1", 0), Refuse) + threading.Thread(target=server.serve_forever, daemon=True).start() + free = socket.socket() + free.bind(("127.0.0.1", 0)) + closed_port = free.getsockname()[1] + free.close() + try: + for label, endpoint, status in ( + ("an HTTP 500 handshake", + f"ws://127.0.0.1:{server.server_port}/devtools/browser/x", 500), + ("a closed port", f"ws://127.0.0.1:{closed_port}/x", None)): + loop = asyncio.new_event_loop() + started = time.time() + try: + with redirect_stderr(io.StringIO()): + loop.run_until_complete(engine._connect(endpoint, 15)) + ok &= check(f"[cdp] pyppeteer raises on {label}", False, "connected?") + except engine.CDPConnectError as exc: + ok &= check(f"[cdp] pyppeteer raises on {label}, promptly", + time.time() - started < 10, + f"{time.time() - started:.1f}s") + ok &= check(f"[cdp] and carries the status ({label})", + exc.status_code == status, str(exc.status_code)) + finally: + engine._drain_and_close(loop) + finally: + server.shutdown() + + # The retry wraps it: two 500s, then a browser; a 401 once. + attempts = [] + + async def flaky(endpoint, timeout_s): + attempts.append(endpoint) + if len(attempts) < 3: + raise engine.CDPConnectError("refused", 500) + return "BROWSER" + + saved = (engine._connect, page_flow.CDP_CONNECT_BACKOFF_S) + engine._connect, page_flow.CDP_CONNECT_BACKOFF_S = flaky, (0.0, 0.0) + loop = asyncio.new_event_loop() + try: + try: + with redirect_stderr(io.StringIO()): + got = engine._connect_with_retry(loop, "ws://h/") + except Exception as exc: # report it by name, not as a traceback + got = f"raised {type(exc).__name__}" + ok &= check("[cdp] pyppeteer retries a 500 handshake, then connects", + got == "BROWSER" and len(attempts) == 3, f"{got} {len(attempts)}") + finally: + engine._connect, page_flow.CDP_CONNECT_BACKOFF_S = saved + engine._drain_and_close(loop) + return ok + + def test_run_contract_captcha_budget(): """At most one paid solve per page, whatever retries and rotations do.""" ok = True @@ -2109,6 +2322,9 @@ def main(): test_run_contract_last_page_reason_survives, test_run_contract_bad_gl_is_usage, test_run_contract_output_paths, + test_cdp_connect_policy, + test_cdp_teardown_playwright, + test_cdp_teardown_pyppeteer, test_run_contract_captcha_budget, test_cli_rejects_counts_that_cannot_run, test_env_url_does_not_override_a_typed_builder_flag,