Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ when_to_use: >-
capability:
- capability:platform
- capability:reassess
surface_hash: sha256:7d670b572d43bbeb
surface_hash: sha256:f8a5db3da24024cb
license: Apache-2.0
measured_tokens: 6952
measured_tokens: 6950
---

<!-- Placeholder convention (see AGENTS.md#placeholder-convention-used-in-skill-files):
Expand Down Expand Up @@ -89,7 +89,7 @@ If a fail shows a failure mode not catalogued there, propose appending a new ent
- **Map each βœ— to a catalog entry.** The fail report links directly to the matching section of [`docs/setup/sandbox-troubleshooting.md`](../../../../docs/setup/sandbox-troubleshooting.md).
Do not paraphrase the remediation; the catalog is the single source of truth.

## The 8 probes
## The probes

The probes cover the catalog's failure modes that a sandboxed command can detect on its own; the signing entries that need a live touch or a terminal are verified by `setup-isolated-setup-verify` check 10 instead.
New probes are added when new entries land in the catalog, so the two stay in lock-step.
Expand Down
6 changes: 3 additions & 3 deletions plugins/magpie-setup/skills/isolated-setup-verify/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,9 @@ when_to_use: >-
have started working β€” that is the canary for a denial having
silently become an allow.
capability: capability:platform
surface_hash: sha256:ce33cb9437d44992
surface_hash: sha256:e076930daa2da2cc
license: Apache-2.0
measured_tokens: 9925
measured_tokens: 9923
---

<!-- Placeholder convention (see AGENTS.md#placeholder-convention-used-in-skill-files):
Expand Down Expand Up @@ -99,7 +99,7 @@ Local modifications go in the override file; framework changes go via PR to `apa
`git status` lists such placeholders as untracked (`.bash_profile`, `.claude/settings.json`, `.mcp.json`), and reading one returns nothing, so an empty settings file seen this way is not evidence for checks 1 or 6.
Take the host state from git (`git ls-files <path>`, `git log -- <path>`) or from a read the user runs outside the sandbox, and never delete the placeholder.

## The 12 checks
## The checks

The canonical list lives in
[docs/setup/secure-agent-setup.md β†’ Verification β†’ Via a Claude Code prompt](../../../../docs/setup/secure-agent-setup.md#via-a-claude-code-prompt-1).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Behavioral evals for the `setup-isolated-setup-doctor` skill.
| Suite | Step | Cases | What it covers |
|---|---|---|---|
| `runtime-routing` | Runtime routing | 4 | Codex, Antigravity, Pi and Goose route to their native adapters and never require Claude files |
| `interpret-probes` | Probe interpretation (`## The 8 probes`) | 19 | all-pass, ssh-fail, localhost-fail, docker-skipped, multiple-fail, ssh-skipped-no-env, injection-in-probe-output, signing-key-fail, gh-sandbox-fail, container-gateway pass/not-running/socket-denied/no-backend/relative-CONTAINER_HOST, scratch-on-shared-session-root, dev-tools-not-granted, git-hooks-unreadable, working-dirs glob entry not matched, working-dirs listing the vetted-ops checkout (βœ—) |
| `interpret-probes` | Probe interpretation (`## The probes`) | 19 | all-pass, ssh-fail, localhost-fail, docker-skipped, multiple-fail, ssh-skipped-no-env, injection-in-probe-output, signing-key-fail, gh-sandbox-fail, container-gateway pass/not-running/socket-denied/no-backend/relative-CONTAINER_HOST, scratch-on-shared-session-root, dev-tools-not-granted, git-hooks-unreadable, working-dirs glob entry not matched, working-dirs listing the vetted-ops checkout (βœ—) |
| `after-report` | Report synthesis (`## After the report`) | 5 | all-clear-all-pass, all-clear-with-skips, ssh-fail-with-catalog-link, multiple-fail-two-catalog-links, injection-asks-autofix-rejected |

## Run
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"skill_md": "skills/setup-isolated-setup-doctor/SKILL.md",
"step_heading": "## The 8 probes"
"step_heading": "## The probes"
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Behavioral evals for the `setup-isolated-setup-verify` skill.
|---|---|---|---|
| codex-native | Codex security model | 3 | untrusted or absent guard despite project trust, unsafe vetted write allowance, and the credential read-isolation gap |
| runtime-routing | Runtime routing | 4 | Codex, Antigravity, Pi and Goose route to their native adapters and never require Claude files |
| step-1-classify | The 12 checks | 24 | all-pass, sandbox disabled, missing scripts, version drift, project root missing, injection attempt, signing key unreadable in the sandbox (check 10), dev-tool paths missing from settings.local.json (check 8 ⚠), whole-user hook dir unreadable from the sandbox (check 8 βœ—), working directories missing under the read block with a glob entry (check 15 ⚠), roborev installed with `review` and `show` auto-allowed no config-edit deny and an incomplete `ask` list (check 17b, 17d and 17e βœ—), roborev's hook replacing the shared whole-user dispatcher (check 17f βœ—), `~/.roborev/` made writable with and without step B.2 (check 17c βœ—), a roborev MCP server registered (check 17g ⚠), the vetted-ops checkout listed as a working directory (check 16f βœ—), Linux `allowAllUnixSockets` with no host-socket `denyRead` (check 12e βœ—), `denyRead` missing from every merged scope (check 1 βœ—), settings only in `settings.local.json` beside a sandbox placeholder (check 1 βœ“ naming the file), an IDE-launched session with credential-shaped variable names (check 7 ⚠), a plain `ssh-ed25519` signing key with `commit.gpgsign` on (check 10 βœ“, the overlay sub-checks n/a), a software signing key with an unrelated smart card plugged in (check 10 βœ“, the overlay sub-checks n/a), no sentinel and no `ls` for 7a (check 7 inconclusive), no framework checkout (check 5 ⚠ for the unenforced claude-code floor, checks 10 and 13 unchecked, installed versions still reported) |
| step-1-classify | The checks | 24 | all-pass, sandbox disabled, missing scripts, version drift, project root missing, injection attempt, signing key unreadable in the sandbox (check 10), dev-tool paths missing from settings.local.json (check 8 ⚠), whole-user hook dir unreadable from the sandbox (check 8 βœ—), working directories missing under the read block with a glob entry (check 15 ⚠), roborev installed with `review` and `show` auto-allowed no config-edit deny and an incomplete `ask` list (check 17b, 17d and 17e βœ—), roborev's hook replacing the shared whole-user dispatcher (check 17f βœ—), `~/.roborev/` made writable with and without step B.2 (check 17c βœ—), a roborev MCP server registered (check 17g ⚠), the vetted-ops checkout listed as a working directory (check 16f βœ—), Linux `allowAllUnixSockets` with no host-socket `denyRead` (check 12e βœ—), `denyRead` missing from every merged scope (check 1 βœ—), settings only in `settings.local.json` beside a sandbox placeholder (check 1 βœ“ naming the file), an IDE-launched session with credential-shaped variable names (check 7 ⚠), a plain `ssh-ed25519` signing key with `commit.gpgsign` on (check 10 βœ“, the overlay sub-checks n/a), a software signing key with an unrelated smart card plugged in (check 10 βœ“, the overlay sub-checks n/a), no sentinel and no `ls` for 7a (check 7 inconclusive), no framework checkout (check 5 ⚠ for the unenforced claude-code floor, checks 10 and 13 unchecked, installed versions still reported) |
| step-2-recommend | After the report | 7 | all-pass, install needed, update needed, project-root missing, multiple gaps, roborev findings (settings edit by the operator, `.roborev.toml` left to them), no framework checkout (warn, with a re-run once a clone exists) |

## Run
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"skill_md": "skills/setup-isolated-setup-verify/SKILL.md",
"step_heading": "## The 12 checks",
"step_heading": "## The checks",
"also_include": [
"skills/setup-isolated-setup-verify/conditional-checks.md"
]
Expand Down
9 changes: 3 additions & 6 deletions tools/spec-loop/specs/sandbox-diagnostics.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,7 @@ error, and two skills that probe or verify the setup on demand.
exiting 1 so the line reaches the model and the user. Tests under
`tools/agent-isolation/tests/test_sandbox_error_hint.py`.
- Skill `setup-isolated-setup-doctor` β€” live, read-only probes
(nine, `### Probe 1` to `### Probe 9`, under a section heading that
still reads `## The 8 probes`), each reporting βœ“ / βœ— / ⊘ / ⚠ with the command
(nine, `### Probe 1` to `### Probe 9`, under `## The probes`), each reporting βœ“ / βœ— / ⊘ / ⚠ with the command
and its output as evidence, and each mapping βœ— to the matching
catalog anchor. Each probe is a deterministic, side-effect-free
script, `scripts/probe-<n>-<slug>.sh`, printing one
Expand Down Expand Up @@ -113,8 +112,8 @@ error, and two skills that probe or verify the setup on demand.
which shows what an un-excluded `gh` does on this machine, then
checks that `"gh *"` is configured.
- Skill `setup-isolated-setup-verify` β€” static checks of the installed
configuration (numbered 1–17 under a section heading that still reads
`## The 12 checks`; 13, 14 and 17 optional): settings shape, hook wiring, hook scripts, wrapper,
configuration (numbered 1–17 under `## The checks`; 13, 14 and 17
optional): settings shape, hook wiring, hook scripts, wrapper,
pinned versions, status line, denial canaries, project-root grant
(check 8 β€” also the dev-tool paths, whose absence is ⚠ not βœ—, and in
whole-user scope the shared hook directory probed from inside the
Expand Down Expand Up @@ -293,8 +292,6 @@ PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner \
hook branch beside it" rule; the session-start hook's grant cleanup and `isolated-setup-update`'s
must-fix report are its only other surfaces.
- Probes 2, 4 and 6–9 still have no script tests.
- The doctor's and verify skill's section headings (`## The 8 probes`,
`## The 12 checks`) lag their bodies (nine probes, seventeen checks).
- A command that spells a working directory with a literal `~` still
prompts under the read block even when the directory is listed. For
the vetted-ops commands the setup closes this by requiring the read
Expand Down