Repository navigation
evals: live runner gaps left from #175 (headless turn end, Hermes path-refused, APFS regrade) #204
Description
Split from #175 so its oracle work could close.
- Claude Code headless turn ends while background children still run (
evals/promises/harnesses/claude_code.py:856), soarchitect-runPASSes cover the sketch stage only. - Hermes
architect-runreturnspath-refusedon every run (evals/promises/harnesses/hermes.py:368). - The APFS-clone regrade at the final head is still unrun.
- The
poteto-help-unclear-runcase prompt names the setup situation, so every harness routes instead of asking; the case needs a rewrite (touches the ledger).
Items 1-2 make architect verdicts meaningless on two of four harnesses. Trigger: before the owed #190/#196 live rerun wave.
Activity
mdsmithaustin commentedon Oct 12, 2026
Two more items carried over from #175 (Grok comments at 20:11Z and 20:26Z): (5) Grok's arena run used a fixed
/tmp/arena-import/where candidates read stale files andRUBRIC.md; PR #201 changes arena to a fresh per-run dir deleted at run end, so re-check this on the next Grok wave after #201 merges. (6) the 2400 s per-turn cap note for Grok runs.mdsmithaustin commentedon Oct 15, 2026
Item 4 is resolved by #252. The case now asks for help without naming a setup situation, and the matching ledger description uses the same prompt. Current validation observed clarification across the five existing help topics.
This closes only the ambiguous-case repair. The headless-child lifetime, Hermes, APFS regrade, and broader live verification items remain open.
mdsmithaustin commentedon Oct 16, 2026
Accepted bounded follow-up: keep this issue open for concrete completion-evidence and live verification debt. Focused adapter regression checks pass, but they do not establish that the retained real background children finished or that completion-dependent grading is correct. That Oracle correction remains held pending approved corpus inputs and replay authorization. The ambiguous help case was repaired by #252. Retained Hermes path evidence, APFS regrading, and Grok fresh-scratch observation remain unverified. The 2400-second run remains partial coverage. Defer persistent supervisor architecture; no live wave or replay was launched in this follow-up.