[jehrr]

Problem

There were two remaining holes in how this repo reports a run that finds nothing.

1. --allow-empty destroyed the output it exists to protect. With the flag, a blocked or failed run wrote [] over earlier good output, plus a failed sidecar. This was confirmed by calling finish_run. A genuine empty run was also labelled failed, and diff_runs.py refused to compare it.
2. A final page with no job data fell through to exit 4 completed. This happened in several ways:
   • A small 200 that stayed unreadable through its retries and wait, in state unknown, counted as a read page with zero rows. All three engines then reported an empty result, and so did scraper_api_client.py.
   • An empty payload under an unexplained non-2xx, such as a 400, was accepted as the site's empty answer.

Resulting behaviour

finish_run carries the same patch as woolworths, weibo and wellfound:

Case	Exit	Row files	Sidecar
Genuine empty + --allow-empty	4	[] + header-only CSV	status: empty
Genuine empty without the flag	4	none	none
Blocked / never obtained / parser failure	3 / 5	none; earlier output + sidecar kept byte for byte	earlier sidecar kept; blocked/failed only when nothing earlier exists and the flag is set
Rows then failure	6	rows	partial

diff_runs.py compares an empty run like a complete one.

No job data is exit 4 only when the page is a 2xx and its payload's own container is present and empty (the empty state):

• Served, but no records and no empty container. This covers unknown after its retries, and parse_error. The result is parser_found_nothing: exit 5, or 6 after rows, with the page in pages_unreadable.
• An unexplained non-2xx (anything except 404/410, 403, 429 and 5xx). The result is target_http_error: exit 5, or 6 after rows. The status goes into pages_target_http_error, and the page is not retried. 5xx keeps target_http_5xx and its retries.
• Coverage. This applies in Playwright, Selenium, pyppeteer and scraper_api_client.py. The API client's parse_error now goes through finish_run instead of returning a bare 5.

Evidence

New checks:

• check_allow_empty_writes_only_a_genuine_empty_result: the same scenario set as the siblings. It includes byte-for-byte preservation for blocked, never obtained and parser failure, and the diff_runs acceptance.
• check_no_job_data_is_exit_4_only_when_the_site_says_so: each engine's real scrape() with the browser primitives stubbed. It runs:
  • the site's empty index (200), expecting 4 and empty;
  • an empty index under 400, expecting 5, target_http_error, {"1": 400} and one fetch;
  • a tiny 200 in listings and in job mode, expecting 5 parser_found_nothing;
  • rows then a tiny 200, expecting 6 with pages_unreadable: [2].
• The same outcomes are checked through the real API client main(), with requests.post stubbed.
• In the existing 4xx checks, a blocked --allow-empty sidecar is now expected to say blocked.

Planted-fault controls (Playwright venv). Each one went red, and the suite was green again after restore:

• C1: output_writer and diff_runs from main, 14 failures.
• C2: genuine_empty = not rows.
• C3: genuine empty labelled failed.
• C4: preservation branch disabled.
• C5: diff_runs back to != "complete".
• C6: final unknown counted as read. The engines gave exit 4 completed/single_page_route and exit 0 for rows then no data.
• C7: an empty payload under a non-2xx accepted, giving exit 4.
• C8: the API client treating only parse_error as a failure, giving exit 4.

Suites:

Venv	Checks passed	Failed
core	682	0
playwright	821	0
selenium	748	0
pyppeteer	748	0

ci_checks.py --all was OK.

No paid calls, no proxies and no credentials were used.

Not changed

• Retry-After handling.
• The retry budgets.
• No version bump or release.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CKsd7q25N19WzUZjrAdzx8