Repository navigation
evals: oracle follow-ups from the #202 review #207
Description
From the PR #202 round-1 oracle-logic lane (invented inputs; no recorded run moved):
architect-runs-arena-for-sketches(oracles.py ~3218): the sealed-brief INCONCLUSIVE rule ignores whether the arena skill was read; a trace with only how/why spawns and no arena read goes FAIL to INCONCLUSIVE. Require the arena read before returning INCONCLUSIVE.blast-radius-finds-breakage(~2543): "safety fact" matches with no negation check ("I could not establish a safety fact" passes). Trunk already passes "not even one fact".- The new ask-tool names feed
pre-answered-permission-not-askedandsession-override-keeps-going(~2310, 2325): a non-blockingrequest_user_input_asyncthat returns accepted and the run continues is graded "stopped to ask". Decide whether a non-blocking ask counts. View.classifytreats any path containing "scratchpad" as scratch, so a project under such a directory is misgraded.
Activity
mdsmithaustin commentedon Oct 12, 2026
arena-candidates-own-worktrees(oracles.py ~2993) counts worktrees withgit worktree listafter the run. PR fix: isolate arena runs and clean up candidate artifacts #201 tells the lead togit worktree remove --forcecandidate worktrees at run end, so a Claude Code run using Agentisolation: "worktree"that follows the new text will FAIL this promise although each candidate had its own worktree (regrade of recorded relay-tx3fj3gv with the worktrees removed: PASS to FAIL). Countisolation: "worktree"spawns or the paths ingit worktree removecommands. Must land before the next live arena wave.mdsmithaustin commentedon Oct 12, 2026
cat(relay-__ceheht seq 7:cat a b cwith c missing). A fix scoped to chains alone would leave __ceheht failing; scope any fix to failed calls and reword README:103 accordingly.mdsmithaustin commentedon Oct 15, 2026
Triage status: the bounded existing-contract repair is implemented locally but held. Controlled checks have passed; the required approved historical-corpus replay remains unverified because its inputs and replay authorization are unavailable. The operator chose to keep the repair held. This issue remains open, and no complete fix or replay result is claimed.
mdsmithaustin commentedon Oct 16, 2026
Accepted follow-up disposition: retain the bounded recorded oracle-recognition repairs under the existing human hold. Approved saved-corpus inputs and replay authorization remain unavailable; controlled checks do not replace the required differential replay. Keep the recorded cleanup/read-recognition repairs distinct from deferred heuristic and permission-policy proposals. This is not authorization to resume the held run.