[jehrr]

Problem

The open question: should a Scraper API 404 exit 4 (genuinely not found) or 5 (content not obtained)? The answer depends on whose 404 it is.

• Current 2Captcha Scraper API docs (https://2captcha.com/scraper/scraper-api/api, read 2026-10-15): /tasks/sync with format: "json" returns HTTP 200 with status, http_code, headers and body. The docs say a site response of 404 "is returned with HTTP 200 from the API", and that site 4xx codes "are not considered unavailability". The API's own 404 is "task ID not found" on the async result endpoint, which this client does not call. Its other non-200s are 400, 401, 402, 408, 410, 422 and 503.
• The code reads the target status from http_code (since Scraper API: send waitFor as an object, read the target status from http_code #4), and every non-200 API response raises and exits 5.

So http_code: 404 is Mercor's own 404: the job is gone. The browser engines classify that as not_found from goto()'s status. Driving the real Playwright scrape() with the page stubbed to answer 404 with the sanitized real Mercor 404 body gives exit 4: stop reason completed, and the address in urls_not_found.

The client already exited 4 for this state, but it got there by a different route. It reached "0 jobs parsed", wrote a debug dump and returned 4 with no sidecar, even with --allow-empty. It also never said the job was gone.

Resulting behaviour

• Through the Scraper API, a target 404 is logged as not_found and exits 4 via finish_run. With --allow-empty it writes the same empty output and sidecar as Playwright (stop_reason, pages_completed, urls_not_found, jobs_not_found). Without --allow-empty nothing is written over earlier output.
• A non-200 from the Scraper API itself is still 5.
• A served page with no records also goes through finish_run now (still exit 4), so --allow-empty behaves the same on that path.
• README has a new paragraph under the exit codes, and CHANGELOG [Unreleased] has an entry.

Evidence

• New offline check: check_a_target_404_is_exit_4_in_the_scraper_api_client_as_in_the_browser. It drives the real client from argv with only requests.post stubbed:
  • HTTP 200 with http_code: 404 and the sanitized real 404 body gives exit 4, with a sidecar under --allow-empty and none without it;
  • the same body with http_code: 200 does not give exit 4, so it is the status that decides;
  • API HTTP 404, 422 and 402 give exit 5;
  • the Playwright engine on the same page gives exit 4, and its sidecar agrees with the client's on status, stop_reason, pages_completed, pages_failed, urls_not_found and jobs_not_found.
• Planted-fault controls, each red on the named check and green after restore (PYTHONDONTWRITEBYTECODE=1, caches cleared):
  • the not_found branch returns 5;
  • fetch_html drops http_code;
  • urls_not_found is omitted;
  • the pre-change client (the "with a sidecar" check fails).
• Suite results: core-only venv 629 passed, 0 failed; Playwright venv 728 passed, 0 failed. Engine files are unchanged.
• ci_checks.py --help-check --sample-check --secret-check passes.
• No paid calls, and no key was sent anywhere.

Not changed (found while checking)

A target 5xx (Scraper API status: "error", or a browser goto() 503) classifies as unknown. After the retries it currently ends as exit 4 in Playwright too, verified with the same stub. That is content not obtained and should be 5. It touches all three engines' state handling, so it is left for a separate change.

🤖 Generated with Claude Code