[jehrr]

Fixes from the 2026-10-14 audit (base 29532be, = v0.2.3).

Problem

• A run that never got its content (category page or API down, no catalogue form, off-site form) exited 4 ("zero products"), and the suite pinned that.
• A page inside the category's own total that came back with no rows (renamed row container, records without modelName, an unpainted or unparsed page in a browser engine) ended the run complete, exit 0.
• Outputs were written 0600; no CSV formula guard; no captcha solve budget (up to 3 purchases per page with a 3-exit pool); credential scan read 0 files from a clone under venv/ or captures/; Scraper API sent waitFor as a string, read the verdict string as the status, logged x-debug with a CDP password; private section references in shipped files and the v0.2.3 release notes; another site's URLs in --url help and the bug template.

Behaviour now

• Zero rows: exit 4 only for the site's own "empty / past the end" stop reasons; everything else exit 5 (a rule over the complete set, so a new stop reason cannot fall through to 4). Blocked stays 3; rows then failure stays 6.
• New parse_failed page state (API page empty where expected_records(total, page) > 0) and zero_rows_reason() -> parser_found_nothing / page_not_painted; the page goes into pages_failed. Same in Playwright (incl. the concurrent path), pyppeteer, Selenium and the Scraper API client. Catalogue API refusing with 401/403/429/503 on every attempt -> exit 3.
• Atomic writer chmods the temp file: 0o666 & ~umask for new files, existing mode kept. CSV-only formula escaping, csv_cells_escaped merged under the run's own sidecar keys.
• page_flow.SolveBudget (1 per page) created before the block-retry loop and checked before solve() in all three engines; not reset by rotation.
• ci_checks.py: exclusions on path.relative_to(REPO).parts, .env* backups scanned, 0 files scanned = failure.
• Scraper API: waitFor object, status from http_code (int status fallback), x-debug redacted (URL credentials, key=, JSON secret fields).
• Smaller: API position counts emitted rows; category GET retried --retries times; catalog_client.py --proxy-rotate per-page is a usage error (it was a silent no-op); Playwright/pyppeteer pass goto()'s status to classify (cleared after a solve+reload).
• Canary: exit 3 and exit 5 are now ::error:: (this path needs no secret and served bare runners every day); exit 3 no longer passes as a notice.
• pip-audit: PYSEC-2026-4175 / -4177 (urllib3 1.26.20, fixed only in 2.8.0) added to the pyppeteer ignore list by ID; pyppeteer 2.0.0 declares urllib3>=1.25.8,<2.0.0, so no compatible pin exists. README updated (seven advisories).

Evidence

• Core-only venv: smoke_test.py 577 passed, 0 failed, 8 skipped. Selenium venv: 588/0/7. pyppeteer venv: 588/0/7. Playwright venv (no browser download): 606 passed, 2 failed — both in check_fingerprint_refusal_is_an_api_error, which launches Chromium and fails locally only because no browser is installed; CI's engine job installs one.
• 16 planted-fault controls (PYTHONDONTWRITEBYTECODE=1, __pycache__ cleared), each red on its named check and green after restore: exit-5 rule, parse_failed state, Playwright or not products fold, chmod removed, CSV escape off, Selenium budget bypassed (first version of that check stayed green against if False and not budget.spend(); rewritten as an AST check), absolute-path exclusion, .env.bak not scanned, waitFor string, x-debug verbatim, http_code ignored, citation planted, position by slot, dropped records -> complete, API 403 not a block, per-page accepted.
• Behavioural probes with stubbed drivers: solver invocations for one page with a 3-exit pool = 1 in Selenium, pyppeteer and Playwright (audit measured 3). Selenium scrape(): page 2 unpainted -> exit 6 page_not_painted; page 2 content/0 parsed -> exit 6 parser_found_nothing; page 2 exhausted -> exit 0 complete; page 1 parsed nothing / load failed -> exit 5.
• Real data: the 2026-10-14 canary's four API payloads replayed through the new catalog_client -> 48 rows, complete, exit 0 (no false parse failure on the real last page). One free live run (--pages 10, 2026-10-15): 48 products, 4 pages, complete, csv_cells_escaped: 0, files 0664 under umask 002.

Not changed

• Version bump, tag, release (entries are under [Unreleased]).
• Shared fetch-loop refactor, schema/sinks (decisions, not fixes).
• --solve-captcha always on a page that is not blocked still buys a token and leaves it in the page's own field (explicit opt-in; unchanged).
• Retired-advisory policy for pyppeteer (documented and ignored by ID rather than dropping the engine — owner may prefer otherwise).
• Selenium cannot provide an HTTP status; it still classifies from the document.

Metadata/releases (outside the diff): v0.2.3 release notes had two "template §NN" citations removed, rest byte-identical; topics: removed price-monitoring (the dataset has no prices), added product-catalog and scraping-browser.

🤖 Generated with Claude Code