[mdsmithaustin]

Why

Recent promise evaluation receipts left Claude Code unrun because preflight rejected its re-signed ripgrep. This report records the rerun after admission support landed.

What changed

• Add the October 24 Claude Code results for eight live cases: nine PASS verdicts and one INCONCLUSIVE verdict across nine promises.
• Document coverage gaps, the swarm environment failure tracked in promises: swarm-run fixture check.sh fails with no dirname in the Claude Code sandbox #309, and comparisons with earlier receipts.

Scope

Only adds evals/promises/results/2026-10-24-claude-code-rerun.md. Other agents and the deferred verify-skill-run case remain outside this rerun.

Blast Radius

Documentation only, affecting readers of the promise evaluation results. No runtime behavior changes.

Verification

The report records eight live cases with no grader refusals or preflight errors. Two turns hit the 300-second cap, and their PASS verdicts stand because the routed skill loaded before the cap. One verdict still requires judge review; swarm checks were blocked by a missing dirname command. No checks were rerun during PR drafting.

✅ Low: The change adds one historical Markdown receipt with no executable effect. Its claims match the ledger and case bindings. The only issue is that its case and verdict counts are newer than the intent text.

Updates from git push no-mistakes

[mdsmithaustin]

Verdict: FAIL
Head: de84ea7
Patch-id: 2f051a598eb109fbcb4cb86498b24c86a5703613
Docs: needs changes

Trail reviewer (opus@xhigh), read-only. I re-checked the head and base with gh and git ls-remote. The head is de84ea7a and main is at d7ae9199. Every verdict in the matrix is right. Two claims in the receipt are wrong, and one coverage claim is incomplete.

1. "Every turn exited 0 without a timeout" is false. Two turns hit the 300 s case cap with exit -9. They are name-only-route-figure-it-out (b/relay-qtf7rne_, turn 0, 300.0 s) and name-only-route-after-compact (b/relay-guy1996m, turn 2, 300.0 s). Both run.json files record timed_out: true, and both traces have x_timed_out: true and x_evidence_complete: false. The two PASS verdicts still stand, because in both runs the lead loaded figure-it-out through the Skill tool at seq 8, early in the turn. The 2026-10-09 and 2026-10-10 receipts record the same pattern as "Every load happened before the timeout, so the verdicts stand." Replace the sentence with the two timeouts and that reason. The PR body's "exited successfully without timeouts" has the same error.
2. The changed-owner list misses a feat(pr-recap): post a story-first reviewer recap with a hand-drawn sketch #300 owner. feat(pr-recap): post a story-first reviewer recap with a hand-drawn sketch #300 (5048ec5c) also added the Recap line to skills/poteto-mode/playbooks/opening-a-pr.md, and fix(pr-recap): refresh the recap after any push and skip major pin bumps #303 edited that file again. At d7ae9199 the ledger names that file as an owner of four live promises. open-pr-prompt-routes-to-opening-a-pr is bound to the runnable route-open-pr case (300 s). The other three are bound only to deferred cases: opening-a-pr-worktree-small-commits-link (open-pr-run), opening-a-pr-gated-publish-never-bypasses (open-pr-gated-run), and worktree-request-honored (worktree-isolation-run). The receipt says "This run covers the promises whose owners changed in feat(pr-recap): post a story-first reviewer recap with a hand-drawn sketch #300, sync: upstream eb4db2b (statechart skill, #741) #297, and sync(upstream): port eb4db2b..e73df47 (statechart update, ketchup-video, script fixes) #304" and names none of the four. No receipt since feat(pr-recap): post a story-first reviewer recap with a hand-drawn sketch #300 has run route-open-pr. The gap starts at line 7 of the eb4db2b receipt, which lists only two feat(pr-recap): post a story-first reviewer recap with a hand-drawn sketch #300 owners. Either run route-open-pr on Claude Code at d7ae9199, or add the file and its four promises to "Changed live owners covered" and list route-open-pr as not run, next to the deferred note for swarm-splits-verification-by-feature-map.
3. The dirname error is quoted from the wrong call. Each worker's check.sh run printed <checkout>/packages/<pkg>/check.sh: line 3: dirname: command not found, then NO TESTS RAN and exit=5. The worker transcripts are agent-aac561aef2bdf9749 (ingest), agent-a20ef238ff0e6307b (publish), agent-abef766ffa3dc5b51 (render), and agent-afc115a991bd2a6ef (shape). The string /bin/bash: dirname: command not found comes from the lead's own worktree-setup Bash call, next to /bin/bash: rmdir: command not found. The rest of the bullet matches the trace. Quote the check.sh: line 3 text instead.
4. For promises: swarm-run fixture check.sh fails with no dirname in the Claude Code sandbox #309, the cause is the Seatbelt read allowlist, not PATH. The worker PATH includes /usr/bin, and ls /usr/bin/dirname printed Operation not permitted. The sandbox-exec profile in run.json denies file-read* and then allows a literal list of binaries that has no /usr/bin/dirname. The issue's second fix, "the adapter's confined PATH, so it includes coreutils", would not reach dirname. The fixture change (cd "${0%/*}") or a wider read allowlist would. This note is for the issue. The receipt needs no change for it.
5. The INCONCLUSIVE swarm row reports the oracle's verdict. c/kiln-vuz3gmm0/verdict.json records INCONCLUSIVE with needs_judge: true and the evidence table separators: 1, gaps mentioned: False, and all four packages covered. The receipt keeps INCONCLUSIVE and labels its PASS reading as a hand reading, which is correct. One addition would help a reader. swarm_report in evals/promises/oracles.py passes a reply that has a table and matches \bgaps?\b|dropout|none missing|no gaps. This reply has the table but none of those words, so the oracle sent it to the judge. That fact explains the move from the 2026-10-03 PASS better than "leaves compactness and gap reporting to a judge".
6. The swarm trace also has x_evidence_complete: false. The lead started all four workers with run_in_background: true (seq 17, 19, 21, and 23), and evals/promises/README.md line 252 marks that as incomplete evidence. With note 1, three of the seven traces are incomplete. No oracle reads the field, so no verdict changes. State it so a reader knows which traces are partial.
7. The 2026-10-03 baseline needs the PORTING.md caveat. PORTING.md step 8 says 2026-10-03.md measures the tree before its own fixes, so its "Fixed in this stack" rows are the baseline. The e73df47 receipt says this and notes that the section names no swarm fix. This receipt leaves that out. The 2026-10-03 screen also ran before fix(promises): confine Claude native filesystem access #138 added the Seatbelt confinement on 2026-10-05, so its check.sh scripts ran. This run reached only BLOCKED. Add both facts to the swarm comparison.

What checks out:

• Matrix. All seven verdict.json files agree with the receipt row for row. live.py report on copies of a, b, and c printed 8 promises and 9 verdicts, PASS 8 and INCONCLUSIVE 1. The "not run" cells and the "(judge)" tag follow the e73df47 receipt's style. The run folders hold no refusal file, and every case got past preflight.
• Bindings. Each case's case.json promise list matches the ledger. swarm-splits-verification-by-feature-map is bound only to verify-skill-run, which is deferred for the reason the receipt gives.
• Examined revision. Every run.json and verdict.json records skills_at d7ae91995a8b1aefab27658b8caefaaf90c0dd5b. The grade-controller copies of oracles.py and grade_boundary.py are byte-identical to d7ae9199. launch.json records Claude Code 2.1.307 with SHA-256 5b113a7752119470f1da419cb0e608af3361cae25b614216b13022093958fb2a. Both PR commits change only the receipt.
• Baselines. The 7a9a3dd matrix has Claude Code PASS for the four skill-listing promises and PASS (2) for poteto-help-finds-need-and-routes. The 2026-10-03 matrix has Claude Code PASS for all three swarm promises. No later receipt has Claude Code rows for these promises.
• Issue promises: swarm-run fixture check.sh fails with no dirname in the Claude Code sandbox #309 is open and describes this run.
• Gates at the head. In a detached temporary worktree, python3 evals/promises/ledger.py audit exited 0 with 0 problems, 962 units, 264 promises, and 24 deferred. tools/gate-check.sh lint exited 0 with 8 of 8 checks passing. CI is green.

Docs: I ran documentation-impact in review mode on d7ae9199...de84ea7a. The receipt is the only changed file, and no index or guide lists receipts, so no other doc needs an update. Notes 1, 2, and 3 are wrong or incomplete claims in the receipt itself, so the docs verdict is needs changes.

[mdsmithaustin]

Verdict: PASS+NOTES
Head: 89ed235
Patch-id: f82b6863fc49c053c9573c3d920ac8f015e1aa66
Docs: pass

Trail reviewer (opus@xhigh), read-only re-verification. This comment replaces my earlier FAIL on de84ea7a. I re-checked the head and base with gh and git ls-remote. The head is 89ed2352 and main is at d7ae9199. Commit 89ed2352 changes only the receipt. Every correction matches the run output, and every claim I checked is now true. The notes below are small additions, not blockers.

1. The new open-pr-prompt-routes-to-opening-a-pr row has no baseline comparison. The 7a9a3dd matrix (line 48) has Claude Code PASS for it, so this run matches. The 7a9a3dd bullet in "Comparison baseline" still lists five promises and says "This run matches all five". Add the sixth promise to that bullet so every row in the matrix has a baseline, as PORTING.md step 8 asks.
2. The "Fixed in this stack" sentence leaves out one fact. The receipt says to read the "Fixed in this stack" rows of 2026-10-03 as the baseline, then compares against that file's matrix rows. The section names no swarm fix, as the e73df47 receipt says. Adding that clause tells a reader why the matrix rows are still the nearest swarm evidence.
3. The three deferred-only opening-a-pr.md promises are counted but not named. They are opening-a-pr-worktree-small-commits-link (open-pr-run), opening-a-pr-gated-publish-never-bypasses (open-pr-gated-run), and worktree-request-honored (worktree-isolation-run). The receipt names swarm-splits-verification-by-feature-map in the same situation. Naming the three would make the receipt match itself.
4. The PR body still says the cases "exited successfully without timeouts". The coordinator plans to edit it, so the receipt verdict does not depend on it. The body needs that edit before it is accurate.

How each correction checks out:

• N1, timeouts. b/relay-qtf7rne_ turn 0 and b/relay-guy1996m turn 2 each record exit_code: -9, timed_out: true, and 300.0 s. Both traces have x_timed_out: true and x_evidence_complete: false, and both verdicts cite the figure-it-out load at seq 8. The swarm-run trace has x_evidence_complete: false because its four workers ran with run_in_background: true. No oracle in evals/promises/oracles.py reads that field.
• N2, the new run. d/tally-b1wgqkwt ran route-open-pr at skills_at d7ae91995a8b1aefab27658b8caefaaf90c0dd5b on Claude Code 2.1.307, with the same SHA-256 as the other runs. Turn 0 exited 0 after 30.5 s with timed_out: false, and the trace has x_evidence_complete: true. The verdict is PASS with "first playbook or target skill read by the lead: opening-a-pr". The lead's first tool call read task-triggers.md and then the opening-a-pr.md playbook, and that playbook is the only one in the trace's files_read. The grade-controller copies of oracles.py and grade_boundary.py under .d-grade-controller are byte-identical to d7ae9199. live.py report on copies of a, b, c, and d printed 9 promises and 10 verdicts, PASS 9 and INCONCLUSIVE 1, matching the receipt row for row. The run folders hold no refusal file. The owner line for opening-a-pr.md matches the ledger at d7ae9199.
• N3, dirname quote. All four worker transcripts print check.sh: line 3: dirname: command not found, then NO TESTS RAN and exit=5.
• N5, INCONCLUSIVE reason. "uses none of the oracle's gap words" matches swarm_report and the recorded evidence gaps mentioned: False. The row still reports the oracle's INCONCLUSIVE.
• N7, baseline caveat. PORTING.md step 8 is cited correctly. fix(promises): confine Claude native filesystem access #138 (7bad53e2, 2026-10-05) first added sandbox-exec to the Claude Code adapter, after the 2026-10-03 screen.
• N4. The 2026-10-24T19:26Z comment on promises: swarm-run fixture check.sh fails with no dirname in the Claude Code sandbox #309 names the sandbox's file-read allowlist as the cause.
• Gates at the head. In a detached temporary worktree, python3 evals/promises/ledger.py audit exited 0 with 0 problems, 962 units, 264 promises, and 24 deferred. tools/gate-check.sh lint exited 0 with 8 of 8 checks passing. On GitHub, CodeQL, the three Analyze jobs, skill-checks, and worktree-audit pass. The skills job was still running when I posted.

Docs: I ran documentation-impact in review mode on d7ae9199...89ed2352. The receipt is the only changed file, and no index or guide lists receipts, so no other doc needs an update. I found no false claim in the receipt. Notes 1 to 3 are optional completeness edits.