[jehrr]

Problem

#7 and #8 settled 404 (exit 4) and 5xx (retry, then exit 5 or 6). The other 4xx statuses from Mercor were not handled consistently:

• 410. It was not recognised. The page was retried and ended as exit 5 in every engine and the API client, or exit 4 if its body held an empty payload.
• 429. It was not recognised either. A 429 whose body had no payload was retried as unknown, then ended as exit 4 with stop reason completed: a throttled run claimed an empty result.
• 403. It was already blocked (exit 3), but the stop reason differed by engine. Playwright said blocked_refused; Selenium and pyppeteer said blocked_cloudflare (hard block). scraper_api_client.py returned a bare 3 without going through finish_run.

Resulting behaviour

The same in Playwright, Selenium, pyppeteer and scraper_api_client.py. The API client reads the target status from http_code.

Target status	State	Retried	Final
404, 410	not_found	no	exit 4, counted in jobs_not_found
403	blocked	within the block-retry budget (another exit with a pool, one re-fetch without)	exit 3, or 6 after rows; blocked_refused
429	throttled (new)	the same budget, with a pause starting at twice --retry-delay, doubling, capped at 60 s; never solved	exit 3, or 6 after rows; throttled
5xx	unchanged from #8

A page that carries its records is still content, whatever the status.

In scraper_api_client.py:

• The same doubling pause applies between its --retries attempts for a 429.
• A 403, or a 429 that persists, now goes through finish_run. Earlier output is kept, and with --allow-empty the sidecar names the stop reason.

Evidence

New checks:

• check_the_target_4xx_statuses_classify_by_what_they_say: the classifier on real fixtures (NOT_FOUND_HTML, a captured 404 page) and on a small refusal body; a fixture with records stays content; the pause schedule and its cap.
• check_target_4xx_statuses_agree_in_every_engine: each engine's real scrape() and _fetch_one_page, with only the browser primitives stubbed. For 410, 403 and 429 it checks:
  • the exit code;
  • the number of fetches;
  • the sidecar's stop reason;
  • that the throttle pause is used for 429 only;
  • that earlier output is kept;
  • exit 6 after a good page;
  • that a 429 which clears gives exit 0.
• check_target_4xx_statuses_in_the_scraper_api_client: the real main() with requests.post stubbed. It checks the exit code, the number of attempts, the pauses ([2.0, 4.0] for 429 against [1, 1] for 403) and the sidecar.

Planted-fault controls (run in the Playwright venv). Each one went red on the named check, and the suite was green again after restore:

• C1: 410 not mapped. Engines gave exit 5 with 2 fetches; the API client gave exit 5.
• C2: 429 not classified. Exit 4, completed, which is the old behaviour.
• C3: throttled not counted as blocked.
• C4: Playwright used a linear pause for 429.
• C5: the API client used a flat pause.
• C6: the API client returned a bare 3.
• C7: the throttled stop reason was dropped.

The Selenium and pyppeteer runs also caught the cloudflare (hard block) label before it was unified.

Suites:

Venv	Checks passed	Failed
core	662	0
playwright	795	0
selenium	722	0
pyppeteer	722	0

Engine libraries were installed without browser downloads. ci_checks.py --all was OK.

No paid calls, no proxies, no credentials and no live requests were made.

Not changed

• Retry-After is not read. Selenium exposes no response headers, and the pause schedule is the same in every engine.
• This repo's finish_run keeps its existing --allow-empty behaviour. A zero-row sidecar says failed whatever the cause; the stop reason says which. A blocked run with --allow-empty can still write an empty file over earlier output. The sibling repos' --allow-empty patch would cover it.
• No version bump or release.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8