[jehrr]

Fixes from the 2026-10-14 family-rules audit of main @ 59227a0.

Behaviour changes for existing users

• A run that never obtained its page (load timeout, dead proxy, unknown stop reason) now exits 5, not 4. Exit 4 is kept for a served page that genuinely lists nothing.
• A 404/410 address is stop_reason: not_found, exit 5. Before this, Playwright called it blocked (exit 3, retried on every exit) and Selenium/pyppeteer called it an empty category (exit 4).
• Output files get 0o666 & ~umask and a replaced file keeps its mode. Before this they were 0600 under every umask.
• Negative --delay, --retry-delay, --proxy-block-retries and --proxy-sessions are now exit 2, with an error that names the flag. So is --timeout below 1 in scraper_api_client.py.
• CSV cells that start with = + - @ TAB CR LF get a leading '. This applies to CSV only; csv_cells_escaped in the sidecar counts them.
• The canary goes red on exit 3, 4, 5 or 6. It no longer skips green on a block.

What changed, with evidence

Finding	Fix	Evidence
F1–F3 Scraper API	waitFor is now sent as an object. The status comes from http_code; the API's status verdict is only logged. x-debug is redacted before it is logged.	fetch_html with requests.post stubbed: the payload's waitFor is a dict, {"status":"warn","http_code":403} gives 403 (it used to raise TypeError), and the planted ws://u:SECRET@… is absent from the log
F4 exit 5	finish_run: zero rows, not blocked and not complete gives exit 5. A complete reason with pages_failed set is now partial.	page_load_timeout, proxy_failed and an unnamed reason each give 5. A served empty page gives 4. The old check that pinned stop_reason="empty" → 4 (a reason no engine sets) now uses single_page_mode, and a comment explains the change.
F5 captcha budget	page_flow.SolveBudget (SOLVES_PER_PAGE = 1) is created once per page before the exit-rotation loop. Every call site checks remaining(), and the handler calls budget.spend() before solve_recaptcha().	Each engine is loaded from disk with its driver stubbed and given a page refused on 3 exits. Before the fix that cost 3 solver invocations; now it costs 1 in all three engines.
F6 modes	_atomic_write chmods the temp file before os.replace.	New files under umask 022/002/077 get 644/664/600. An existing 0664 file stays 0664.
F7 CSV guard	_csv_escape runs after list joining, touches strings only, and merges csv_cells_escaped under the caller's extra.	Covered by a check
F8 .gitignore	.env* plus !.env.example	git check-ignore: .env.bak, .env.local and .env.save are now ignored, and .env.example is still committable
F9/F19 credential scan	Exclusions use relative_to(REPO).parts. A venv is detected by pyvenv.cfg, and KEY_SHAPED_FIELD tolerates \". The scan fails when it scanned 0 files.	Planted cases: a clone under a venv/ ancestor is now scanned, myenv/pyvenv.cfg is skipped, an escaped apiKey field is reported, and an empty tree fails
F10 typed URL	All four parsers record typed_url before env_config.apply. Typed --category/--site-locale now build the URL.	With GIVENCHY_URL exported, --category skincare --site-locale gb gives /gb/skincare/ in every engine. With no flag typed, the environment value is still used.
F11 not_found	New not_found state for 404/410, or the site's own error page (pageContext … "type":"error","ns":"error") when no status is available. It is terminal and not retried; the engines stop before parsing, and pyppeteer now passes the status.	Live curl on 2026-10-14: /fr/fr/makeup/lips/ and a made-up /us/makeup/<x>/ both return 410 with that page context, and the empty /es/maquillaje/sets-de-regalo/ returns 200 with "ns":"category". The trimmed fixture classifies the same as the full capture.
F12 donor text	Deleted tools/waf_probe.sh and tools/verify_browser_api.sh; both measured transfermarkt and called --mode market-values. Rewrote the issue-template modes, CONTRIBUTING, the review prompt, the selenium/pyppeteer --concurrency message, the scraper_api_client coverage log, and the page_flow/playwright comments.	The vocabulary check now covers .sh/.toml/.example. Its schema.org/Person word could never match because the text is lowercased first; that is fixed.
F13 private citations	65 lines rewritten in plain words. New test_no_private_citations builds its tokens from pieces.	Release notes for v1.0.0 and v1.0.1 contain no citations, so none were edited.
F16 Dockerfile	Credentials now come from a mounted .env instead of flags
F18 CONTRIBUTING	The exit table now includes 5
Canary (instruction 12)	Removed the exit-3 skip-green path. Every non-zero exit is now ::error::.	No secrets are configured. The last 10 scheduled runs all reached "Sanity-check the scraped data" from a bare runner, so the path needs no secret.
Metadata	Description now reads "2Captcha Scraping Browser API" and "tile microdata and JSON-LD parsing"	gh repo edit

Verification

• Offline suite in a core-only venv: OK, 656 PASS. It also passes in one venv per engine (playwright, selenium, pyppeteer), with that engine's groups run and not skipped, and pip check is clean in each.
• ci_checks.py --all is green.
• 26 planted-fault controls were run with PYTHONDONTWRITEBYTECODE=1 and the cache cleared. Every one turned the suite RED on the expected named check, with no crash, and the restored tree is green. They cover: umask, CSV off, CSV on numbers, extra precedence, the exit-5 rule, pages_failed, waitFor string, status verdict, raw x-debug, the budget call site, a budget reset per exit, the inner budget guard, the no-pool crash, not_found by status / body / engine / sac, typed URL, negative delay, absolute-path scan, venv by name, zero-scan, escaped JSON key, .env*, citation reintroduced, and donor vocabulary.
• No paid calls, proxies or credentials were used. The only live requests were free curl GETs to givenchybeauty.com.

Not changed

• F14 (pyppeteer hint-less UA, page.authenticate, CDP ignoreHTTPSErrors) needs a live run through a proxy or CDP endpoint.
• F15 (--pages >1 is reported and ignored rather than refused) and F17 (meaning of scraper_api_client --retries) are owner decisions.
• No version bump, tag or release (T13); the changes sit under [Unreleased].
• The shared fetch loop and schema/sinks (T10/T11) are out of scope.
• .env* backups are still not scanned, because they are now gitignored and the scan skips ignored files by design.

🤖 Generated with Claude Code