Repository navigation
fix(promises): stop fixture scripts calling dirname in the Claude Code sandbox - #313
Conversation
mdsmithaustin
changed the title
|
Verdict: FAIL Root verdict from a full swarm at this head: one lane that ran things (sonnet@high) and one audit lane (sonnet@xhigh, stepped down from the opus work model). Neither took the PR body as fact. The code is clean. Only the receipt text needs changes. Live and gate lane: PASS.
Audit lane:
Findings sent back (receipt text, in scope, a reader reaches them):
Note (non-blocking): line 36 says the oracle matched the reply's "Gaps: none" line. The regex |
166e761 to
33190c0
Compare
|
Every
Not covered. Codex, Hermes, and Grok did not rerun, and the three 2026-10-24 relay rows that ran against the old Details for reviewersRisk. Low. The diff touches only eval fixtures, one test class that runs on the measured macOS host, a README paragraph, and two eval receipts. No skill or adapter code changes. Base Start here. kiln check.sh and Then.
Proof. CI on the head passes. These pointers are a start, not the review. Read the whole diff, because a defect can sit in a file this recap ranks low. |
The kiln check.sh scripts exit 5 with NO TESTS RAN under the sandbox, and relay's old_callers.sh prints 0 instead of 3, because the read allowlist denies /usr/bin/dirname. This test fails until the fixtures stop calling dirname.
The Claude Code sandbox's read allowlist denies /usr/bin/dirname, so every swarm-run worker's check.sh ran from the checkout root and reported BLOCKED. The scripts now strip the file name from $0 in the shell, and keep the current directory when $0 has no slash, as dirname did.
…eipt Add the open-pr routing promise to the 7a9a3dd baseline, say why the 2026-10-03 matrix rows stay the swarm baseline, and name the three opening-a-pr promises bound only to deferred cases.
…ure fix All four workers now run check.sh and report PASS or ISSUES, and the three swarm promises grade PASS.
Review found that the parity check passed when both runs broke the same way. The test now covers every fixture script, runs each by absolute path and by bare name, and requires the known exit code, stdout, and unittest summary inside and outside the sandbox.
Say that the gap-reporting PASS rests on a no-gap reply, record the lead's mktemp, dirname, and rmdir denials as open, call the 2026-10-03 rows context rather than a baseline, and name the run, its models, and the fixtures tree that survives the squash.
…eipts The three 2026-10-24 name-only relay leads ran old_callers.sh against the broken script and got 0. Their PASS verdicts stand because no oracle reads the count, and the rows were not rerun. The open skill-command denials point to #316, and the gap-oracle sentence names the heading the pattern matched first.
33190c0 to
63ba227
Compare
|
Verdict: PASS+NOTES This root verdict for round 2 replaces the round-1 FAIL at A single trail reviewer (sonnet@xhigh, stepped down from the opus work model) verified The rebase. I checked it myself. Round-1 findings, checked against the saved 2026-10-24 traces:
The gaps sentence now names the heading match at offset 1079 that comes before the "Gaps: none" bullet at 1099. Notes (non-blocking):
|
Why
In the 2026-10-24 Claude Code rerun, every
swarm-runworker reportedBLOCKED. The Claude Code sandbox denies reads of/usr/bin/dirname, so each fixturecheck.shran from the checkout root, found no tests, and exited 5. This PR removes the fixtures' need fordirnameand reruns the case. It also applies the three verifier notes from #310 to the 2026-10-24 receipt.Closes #309
What changed
kilncheck.shscripts andrelay/tools/old_callers.shnow take their directory from$0in the shell. Under the sandbox,old_callers.shprinted0instead of3, and it now prints3.NativeFixtureScriptsruns every fixture script inside and outside the sandbox and requires its known exit code, output, and unittest summary. A first version lands before the fix and fails on the old scripts. Review then made it check each script's known result. It runs only on the measured macOS host, so CI skips it.swarm-runrerun. The 2026-10-24 receipt gains the docs: record Claude Code promise evaluation rerun #310 notes, a pointer to the new receipt, and review corrections to its tense, the fix(promises): confine Claude native filesystem access #138 merge date, and the 2026-10-03 baseline wording. Both receipts now say that the three 2026-10-24name-only-*relay leads ranold_callers.shbefore the fix and got0. Their PASS verdicts stand because no oracle reads that count, and those rows were not rerun.Scope
swarm-runcase only. Codex, Hermes, and Grok did not rerun, because the fix does not depend on the harness.mktemp,dirname,rmdir). The receipt records them as open, tracked in promises: skill scripts call coreutils the Claude Code sandbox blocks (log.sh, portable paths, mktemp) #316.Tradeoffs
Adding
/usr/bin/dirnameto the adapter's read allowlist would also fix #309. It would widen the sandbox for one tool and leave the fixtures depending on a binary that other sandboxes may deny too.Blast Radius
Eval fixtures, one host-gated test, one README paragraph, and two eval receipts. No skill or adapter code changes. The new test runs only on the measured macOS host, like the
NativeGittests.Verification
python3 -m unittest test_claude_code.NativeFixtureScriptspasses on the fix. On the old scripts it fails 6 subtests (kiln exit 5 instead of 0 or 1, relay0instead of3). With an unguardedcd "${0%/*}"it fails 8.live.py run --harness claude-code --case swarm-run: workers rancheck.shand reported PASS foringest,publish, andshape, and ISSUES forrender. All three swarm promises graded PASS. The gap-reporting PASS comes from a run with no gaps, so it is weak evidence, as the receipt says.ledger.py auditreports 0 problems,tools/gate-check.sh lintpasses, and the promise suite under the pinnedpython3.14passes 844 tests. An earliertools/gate-check.sh testrun passed every tier except the statechart node tests, which fail 18 of 146 on this host onmainas well. A three-arm interrogate review led to the test and receipt corrections.✅ Low: The change edits one line in each of five fixture scripts and adds a test on the measured host only. The test checks real behavior inside and outside the sandbox against expected results known independently, and would have failed before the fix through the absolute-path run inside the sandbox. The receipt claims I checked against git (fixture tree hash, skills/ tree identity) hold.
Updates from git push no-mistakes