Repository navigation
fix: isolate arena runs and clean up candidate artifacts - #201
Conversation
|
Verdict: FAIL Round 1 swarm at this head: gates + live scratch + regression (sonnet@high), upstream-parity review and consumer review (opus@xhigh each; work model sonnet@high). Gates: PASS (lint 5/5, skill-ci lint and check 0 findings, ledger audit 0 problems; only known host noise). Live: each run gets a distinct 0700 dir, no RUBRIC.md reachable; the new retired-text rule fails on trunk's arena text and matches nothing else. poteto-help: head wording yields 200 links where trunk's produced the 404. Fix-forward (in scope):
Routed elsewhere (not this PR): the eval oracle grades a graft under the run's own |
347988b to
0e7bd82
Compare
mdsmithaustin
changed the title
|
Verdict: FAIL Round 2 swarm at this head: gates + literal two-run replay + upstream diff + merge simulation (sonnet@high), fix review and consumer review (opus@xhigh each; work model sonnet@high). This is the second FAIL; a third stops the PR. Item 1 (upstream #538 step-3 wording): fixed. The Phase F to E graft carve-out is unambiguous. Gates pass; a run that follows the text leaves nothing for the next run in the mktemp branch; no oracle or harvest reads arena's scratch; merges clean with #202 and #203. Fix-forward (in scope; the leak fix still claims more than the text does):
Merge order: #201 lands immediately before #202 (#202's README describes #201's path; #201 alone misgrades a run root outside the default temp paths, which no recorded run uses). |
0e7bd82 to
97b3866
Compare
mdsmithaustin
changed the title
|
Verdict: FAIL Round 3 swarm at this head: gates + independent 24-scenario two-run replay + oracle checks (sonnet@high), directive review and consumer review (opus@xhigh each; round-3 work on opus@high, so these may be same-family reviews). This is the third FAIL. Per the autopilot playbook the owner is stopped and the PR goes to the operator. What holds: gates pass; across 24 replayed scenarios no candidate, worktree, arena branch, judge score or scratch dir of run 1 reaches run 2 (main leaks on every branch); git refuses to delete the main checkout or a branch checked out elsewhere, so cleanup cannot delete shipped code or the user's branch; round-2 items 1-4 fixed on Claude Code, Codex and Hermes; merges clean with #202, #203, #205. Blocking (recorded run reaches it): Grok Build isolates candidates as standalone repo copies, which |
|
Fail-count reset: the operator approved one more round in chat on 2026-10-12, scoped to making candidates real git worktrees on every harness and proving it with a live Grok arena run. |
97b3866 to
3b2f6d6
Compare
mdsmithaustin
changed the title
mdsmithaustin
changed the title
|
Verdict: PASS+NOTES Round 4 (operator-granted final round; candidates as real git worktrees on every harness) at this head: gates + independent replay + recorded live Grok evidence (sonnet@high), directive review and documentation-impact / consumer review (opus@xhigh each; work on opus@high, so these may be same-family). The round-3 blocker is fixed. Recorded paid Grok run relay-rij6hlqz (at cecfe487, procedure word-for-word identical to this head) passed 5/5: the lead made five detached git worktrees itself, skipped Grok isolation, and removed all of them; no Grok clones were made. An independent replay (84 checks, four isolation scenarios, two consecutive runs) found no run-1 leak and only main left each time. The common-dir check holds across /var vs /private/var, case differences and worktrees of worktrees. Gates pass; CI 7/7; merges clean onto main 5c40398 and with #203, whose PORTING arena rows then match the arena text. Notes (follow-ups, #219): the documentation-impact lane flagged opening-a-pr.md:24's "link the arena directory" now that arena deletes it (the agent finds the directory gone, a loud and recoverable miss, so it is a note here); create worktrees one at a time; pass |
… run Closes #194. Arena's intro and Start list collapse to upstream's one worklist line, and Phase C takes upstream's wording. Phase A step 4 replaces the fixed /tmp/arena-<slug> fallback with a mktemp -d made per run and keeps the rubric outside candidate-readable paths. A retired-text rule bans the fixed path. poteto-help now says to link the repository path, such as skills/how/SKILL.md, so the printed links resolve.
…tep 3 ending
Phase A step 3 ends with upstream #538's wording ("Spawn more runners when",
"Use the same model N times when"). Step 4 now deletes the run's scratch
directory when the run ends, after the synthesized artifact and note are saved
outside it, and before any re-run from Phase E or F. Without that, a later run
could still list and read an earlier run's sibling arena-<slug>.* directory.
The PORTING note now claims only what the text prevents.
"Save the synthesized artifact and synthesis note outside it" named no place, and a Codex lead kept synthesis.md in its run folder. Outputs now puts the artifact where the calling task says, or in a fresh mktemp -d outside the project, and gives the synthesis note in the reply. The preferred git worktree branch was never cleaned up, so the next run saw earlier candidates through git worktree list. Cleanup now removes each candidate worktree and its branch as well as the scratch directory, and Phase E and Phase F repeat the rule where the lead re-frames or finishes. PORTING claims only what this prevents for a run that completes and names what it does not cover (#209).
Grok Build's isolation: worktree makes a standalone clone under ~/.grok/worktrees/, which git worktree list omits and git worktree remove refuses with exit 128, so arena's cleanup left every Grok candidate behind (grok arena-run relay-go6dzbwe left five). Arena step 4 now counts a native isolation option only when git rev-parse --git-common-dir on its path prints the project's. Every other candidate worktree is made with git worktree add --detach inside the run's mktemp -d, so cleanup is the same on every harness. pstack-harness states the same rule and corrects its Grok Build row. Arena's Outputs keeps a sketch awaiting sign-off outside the project (#214 item 2). PORTING's arena note now holds on all four harnesses.
3b2f6d6 to
6456188
Compare
mdsmithaustin
changed the title
Settles the six arena follow-ups from #201 left in #219. Worktrees are made one at a time and Grok writers get isolation none. When git worktree add fails, writers whose output the lead collects get a shared clone detached at each briefed commit. Opening a PR posts the swarm or arena note without findings Shipping keeps off the PR. Closes #219
What Changed
skills/prefix.Validation
Risk Assessment
✅ Low: This is a text-only change to the skill docs, the porting note and the retired-text config. It adds what the intent requires: in arena step 4, the git-common-dir check with --path-format=absolute,
git worktree add --detachin the run's mktemp dir, and spawning on Grok without native isolation. It also corrects the Grok wording in pstack-harness, adds the sign-off rule to Outputs and updates the PORTING note. All recorded human decisions are applied verbatim, and evals/promises/oracles.py is untouched.Testing
The five reference tests passed and the worktree remained clean. The operator reports relay-rij6hlqz covered live behavior with 5/5 PASS; this run exercised no live scenarios and produced no artifacts. Lint was excluded by this phase's explicit prohibition.
Pipeline
Updates from git push no-mistakes
✅ **Intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
git diff 5c40398a31e2d9ed8e1ef2e3d6a81e09db362fe5 6456188818e1b2f7486a5ca8af6817c8617de4f6PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tools -p test_skill_references.py -vConfirmed .arena-validation/, checkpoint-small/, and fallback-repeat/ are absent.Confirmed the change leaves evals/promises/oracles.py and skills/architect/SKILL.md unchanged.git status --shortconfirmed no testing changes remain.✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.