[jehrr]

Problem

When Mercor answered with a server error, detect_page_state classified the page unknown. The engines retried it and then counted it as a served page with no rows. As a result, a run whose only page was a 503 reported an empty result (exit 4), and a 502 inside a --mode job queue was skipped while the run still said complete. The same happened through scraper_api_client.py, where the target's status is http_code of an HTTP 200 Scraper API answer. That path also made only one attempt.

A 5xx says nothing about what the page holds. The requested content was not obtained.

Resulting behaviour

The same rule now applies in Playwright, Selenium, pyppeteer and the Scraper API client:

• Retries: the page is retried within the existing budget (the block-retry loop in the engines, --retries in the client). A 5xx that clears on a retry is a normal run.
• When every attempt is a 5xx: the page is failed, with stop reason target_http_5xx. The run exits 5 if this happens on page 1 and 6 after earlier rows. It never exits 4.
• Sidecar: the status is recorded per page in pages_target_http_error (for example {"1": 503}). The sidecar is written as failed under --allow-empty, and as partial when rows exist.
• Earlier output: with no rows nothing is written, so earlier output and its sidecar are kept.
• Scraper API status: "error" with no target status: exit 5.
• 404: still exit 4 (not_found).
• Shared helpers: target_server_error, failure_stop_reason and target_http_errors live in page_flow.py. They replace seven copies of the inline stop-reason expression.
• Docs: a new README paragraph under the exit codes, and a behaviour-change entry in CHANGELOG [Unreleased].

Evidence

• check_a_target_5xx_is_content_not_obtained_in_every_engine runs each engine's real scrape() and _fetch_one_page, with only the browser primitives stubbed (goto / driver.get + _document_status / pyppeteer goto). It asserts:

  • a 503 on the only page is exit 5, after more than one navigation and within 1 + BLOCK_RETRIES_WITHOUT_POOL;
  • earlier o.json and o.meta.json are byte-identical after the failed run;
  • with --allow-empty the sidecar says failed, target_http_5xx, pages_failed [1], {"1": 503};
  • a good page followed by a 502 is exit 6, partial, {"2": 502};
  • a 503 that clears on the retry is exit 0;
  • a 404 is still exit 4.

• check_a_target_5xx_is_content_not_obtained_in_the_scraper_api_client runs the real client with requests.post stubbed. It asserts:

  • http_code: 503 is exit 5 after 2 attempts, and nothing is written;
  • with --allow-empty the sidecar is written as failed with the status;
  • a 503 followed by success is exit 0;
  • status: "error" with no http_code is exit 5.

• The pre-change code fails both checks: a 503 gave exit 4, and the client made 1 attempt.

• Planted-fault controls. Each was run in its engine's venv with PYTHONDONTWRITEBYTECODE=1 and caches cleared, went red on the named assertions, and was green after restore:

  • Selenium never records the 5xx;
  • pyppeteer omits pages_target_http_error;
  • Playwright's ok ignores server_error;
  • failure_stop_reason loses the 5xx case;
  • the client does not retry the 5xx;
  • the client ignores status: "error".

  One more planted fault, sweeping 4xx into the 5xx rule, stayed green. That fault cannot be reached: a 404 is classified not_found before unknown, and the 404 assertions cover it.

• Suites:

  venv	passed	failed
  core only	638	0
  Playwright	751	0
  Selenium	678	0
  pyppeteer	678	0

• ci_checks.py --help-check --sample-check --secret-check passes.

• No live, paid or credentialled calls were made.

Not changed

• The bounded "unknown" content wait before a retry is unchanged.
• A small, status-less page with no payload (unknown without a 5xx) is out of scope; that path is not a target 5xx.
• No version bump, tag or release.

🤖 Generated with Claude Code