Repository navigation
evals: remaining oracle heuristic gaps after #151 and #156 #175
Description
Follow-ups left after #151 (#115) and #156 (#133) merged. Each item is listed in its PR body with measurements. None is a negated reply that newly passes against main.
evals/promises oracles (#156)
- Review-verdict guard: 122 of 308 correct corpus replies go to the judge (INCONCLUSIVE with
needs_judge). Nothing inreport.pyorlive.pyconsumesneeds_judgeyet, so a person reads those runs. - Label reading (
reply_label): these label shapes read differently from main:- underscore-wrapped and backtick-wrapped labels
- labels ended by a newline
Role:with its value on the next line- a bold prefix before the role
- topic-first headings (
## Parser — Explorer 2) - separator-free
## Explorer 2 Findings - findings-first headings
- Arena F14:
Based on the judge's scores I kept candidate 2 …and markup-wrapped bases (_candidate 2_,[candidate 2](…)) fail where main passed through itsbasesubstring accident. - Sources: commit-only shapes main also passes (comma lists of hashes, consecutive commit-only lines,
Sources searched:labels). - Timing: a
SOURCES_SECTIONwhitespace run is quadratic on pathological tails (same on main). Absolute-pathclassifyis slow on 100k-component paths (same on main).edits()on 200 KB python-bound commands is slower than main. - The
.auditsaved runs were last regraded at round 8. A sandboxed APFS-clone regrade at the final head would measure the real verdict moves.
evals/canon steering-review-asks oracle (#151)
.mock.callson a non-clipboard mock counts as a clipboard read.- A
readText(el)rendered helper is excluded as clipboard. - A fake clipboard bound to an unrecognized constructor counts as rendered, as on trunk.
- Disjoint fragments of one tail fail the present ask.
- The
clipboard_readsfixed point is quadratic on thousands of chained links.
Work each item red-first against main or trunk with a differential corpus, as #151 and #156 did. Several rounds of those PRs regressed because a regex fix was not checked against main in both directions.
Activity
mdsmithaustin commentedon Oct 8, 2026
New oracle and case edges from the post-sync live reruns (PR #179, skills pinned at cf39921). Each was read against its trace. The run evidence is kept locally under
.audit/guide-promises/live/sync-df58112/.Oracle
poteto-help-finds-need-and-routesfails a reply whose example prompt is a 4-space indented code block. It only accepts a triple-backtick fence (hermespoteto-help-run)..../blob/main/poteto-mode/SKILL.md, which dropsskills/and would 404, and the oracle passed the link.ASK_TOOLSomits Codex'srequest_user_input_async.architect-runcounts runners by matching needles in spawn prompts. Codex spawns have encrypted prompts and task names such asdesign_function, so it counted 0 of 7 and failed. The correct verdict there is probably INCONCLUSIVE. Its sketch cell has the same issue.blast-radius-runcounts the agent's ownproof.pyas a file beyond the diff. Its "one fact" regex misses "safety fact". The pasted-command check sends runs to a judge even when they executed real code (claude-code, hermes).Harvest
architect-runcandidates write to the Hermes profile cache scratch, outside the fixture. All three promises come backpath-refused, which will likely recur on every Hermes architect run.Case
poteto-help-unclear-run's prompt, "i just installed this and i'm not sure where to start", names the setup situation.skills/poteto-help/SKILL.md:17sends a named situation straight to its section, so all three harnesses routed instead of asking. The prompt needs a need that is actually ambiguous.Runner
architect-runended its headless turn while a background runner was still running ("The fourth candidate is running..." after 1215 s). Its PASSes cover only the sketch stage.evals(promises): post-sync live reruns for df58112 (#179)
mdsmithaustin commentedon Oct 8, 2026
Two more edges from the Grok cells of the df58112 reruns (evidence under
.audit/guide-promises/live/sync-df58112/grok/):ASK_TOOLSalso lacks Grok'sask_user_question, the same gap as Codex'srequest_user_input_async. It changed no verdict this round.architect-runwrites to the fixed host path/tmp/arena-import/.candidate-2/there still held Oct 4 files from an earlier run, which could leak into the judge step on a run that finishes. Unverified whether any candidate read them. The run also hit the 2400 s cap with no design delivered, so its PASSes cover only the sketch stage, as on Claude Code.mdsmithaustin commentedon Oct 8, 2026
Correction to my 20:11Z comment: the stale-file read is verified, not unverified. The Grok
architect-runtrace (grok/relay-waxsek25/trace.json,x_subagents) shows the "record noun" candidate read the Oct 4/tmp/arena-import/candidate-2/files, and all three candidates read/tmp/arena-import/RUBRIC.md, which the arena says candidates never see. So the shared fixed path already leaked stale and judge-only files into candidates in this run.evals(promises): add Grok cells to the df58112 rerun record (#180)
fix(evals): settle the verdict-moving oracle edges from the df58112 r…
fix(evals): count the run's own tmp directory as scratch when classif…
fix(evals): settle the verdict-moving oracle edges from the df58112 r…
fix(evals): count the run's own tmp directory as scratch when classif…
fix(evals): settle the verdict-moving oracle edges from the df58112 r…
fix(evals): count the run's own tmp directory as scratch when classif…
fix: correct promise evaluation edge cases (#202)
mdsmithaustin commentedon Oct 12, 2026
Closing as split. Verdict-moving oracle edges landed in #202 (321ef77), with the residual heuristic gaps recorded in evals/promises/README.md "Known oracle gaps". Still tracked elsewhere: runner and harvest gaps (headless turn end, Hermes path-refused, APFS regrade, Grok items) in #204; oracle follow-ups in #207 and #208; the arena scratch leak and the poteto-help link wording in PR #201 (stopped at its third FAIL, open items in #214).