Repository navigation
evals: oracles should not FAIL or PASS on a killed run's missing evidence #208
Description
From the PR #198 round-2 semantics lane (policy replays on 11 recorded auth deaths and 1,215 synthetic cut-short runs; scripts under the session scratchpad r2/).
PR #198 keeps every verdict the grader decides after an auth death (policy A), because the alternatives hid real FAILs (recorded relay-yu2tjvlt). The remaining false FAILs come from oracles whose FAIL rests on absence and never check view.killed: step_away (oracles.py ~1123, via overnight_route ~2238), read_only_pin ~1156, routes_and_runs ~1428, principle_steers ~1585, tdd_in_bug_fix ~1655, comment_sicko ~2060, deslop_cleans ~2110, sticky ~2204, trail_reviewer ~2364, unslop_target ~2444, bro ~2459, poteto_help ~2496, doc_impact_before_completion ~2599, doc_impact_review ~2856, doc_impact_modes ~2876, arena_grafts ~3036. Return INCONCLUSIVE at those sites when view.killed, and run completion_gate before anything reading final_reply/reply_of_turn (after a Claude death the 401 text is the final reply). This also covers timeouts.
Mirror problem: interrogate_multi ~1896 and how_narrow ~1682 PASS a killed run because nothing went wrong before it died (relay-yu2tjvlt passed with "reply has a verdict section: False").
Apply the repo's fail-only-on-proof differential gate.
Activity
mdsmithaustin commentedon Oct 15, 2026
Triage status: the bounded existing-contract repair is implemented locally but held with the related #207 repair. Controlled checks do not satisfy the required approved historical-corpus replay. The operator chose to keep the repair held while its inputs and replay authorization are unavailable. This issue remains open, and no complete fix or replay result is claimed.
mdsmithaustin commentedon Oct 16, 2026
Accepted follow-up disposition: retain this as evaluator correctness work under the existing saved-corpus replay hold. The required behavior preserves proved violations as FAIL, treats interruption-caused missing evidence as inconclusive at the affected criterion, and rejects error text as completion evidence. No blanket suppression of killed-run failures is accepted. No replay or publication is authorized by this disposition.