From 06b29b23a8ece5a1a73fc9637be993e0feecf1d6 Mon Sep 17 00:00:00 2001
From: Jarek Potiuk <potiuk@apache.org>
Date: Wed, 28 Oct 2026 01:51:32 +0100
Subject: [PATCH] setup: drop the stale counts from the verify and doctor
 section headings

isolated-setup-verify's checks sat under "## The 12 checks" while the
section numbers them 1-17, and isolated-setup-doctor's nine probes under
"## The 8 probes". Name the sections "## The checks" and "## The probes"
so the headings cannot drift again, and move the eval step configs that
extract by heading, the eval READMEs and the sandbox-diagnostics spec
(which carried the mismatch as a known gap) with them.

Generated-by: Claude Opus 5
---
 .../magpie-setup/skills/isolated-setup-doctor/SKILL.md   | 6 +++---
 .../magpie-setup/skills/isolated-setup-verify/SKILL.md   | 6 +++---
 .../evals/setup-isolated-setup-doctor/README.md          | 2 +-
 .../interpret-probes/fixtures/step-config.json           | 2 +-
 .../evals/setup-isolated-setup-verify/README.md          | 2 +-
 .../step-1-classify/fixtures/step-config.json            | 2 +-
 tools/spec-loop/specs/sandbox-diagnostics.md             | 9 +++------
 7 files changed, 13 insertions(+), 16 deletions(-)

diff --git a/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md b/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md
index fba636995..617ae9678 100644
--- a/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md
+++ b/plugins/magpie-setup/skills/isolated-setup-doctor/SKILL.md
@@ -19,9 +19,9 @@ when_to_use: >-
 capability:
   - capability:platform
   - capability:reassess
-surface_hash: sha256:7d670b572d43bbeb
+surface_hash: sha256:f8a5db3da24024cb
 license: Apache-2.0
-measured_tokens: 6952
+measured_tokens: 6950
 ---
 
 <!-- Placeholder convention (see AGENTS.md#placeholder-convention-used-in-skill-files):
@@ -89,7 +89,7 @@ If a fail shows a failure mode not catalogued there, propose appending a new ent
 - **Map each ✗ to a catalog entry.** The fail report links directly to the matching section of [`docs/setup/sandbox-troubleshooting.md`](../../../../docs/setup/sandbox-troubleshooting.md).
   Do not paraphrase the remediation; the catalog is the single source of truth.
 
-## The 8 probes
+## The probes
 
 The probes cover the catalog's failure modes that a sandboxed command can detect on its own; the signing entries that need a live touch or a terminal are verified by `setup-isolated-setup-verify` check 10 instead.
 New probes are added when new entries land in the catalog, so the two stay in lock-step.
diff --git a/plugins/magpie-setup/skills/isolated-setup-verify/SKILL.md b/plugins/magpie-setup/skills/isolated-setup-verify/SKILL.md
index 92500c8bf..f2f9561cc 100644
--- a/plugins/magpie-setup/skills/isolated-setup-verify/SKILL.md
+++ b/plugins/magpie-setup/skills/isolated-setup-verify/SKILL.md
@@ -16,9 +16,9 @@ when_to_use: >-
   have started working — that is the canary for a denial having
   silently become an allow.
 capability: capability:platform
-surface_hash: sha256:ce33cb9437d44992
+surface_hash: sha256:e076930daa2da2cc
 license: Apache-2.0
-measured_tokens: 9925
+measured_tokens: 9923
 ---
 
 <!-- Placeholder convention (see AGENTS.md#placeholder-convention-used-in-skill-files):
@@ -99,7 +99,7 @@ Local modifications go in the override file; framework changes go via PR to `apa
   `git status` lists such placeholders as untracked (`.bash_profile`, `.claude/settings.json`, `.mcp.json`), and reading one returns nothing, so an empty settings file seen this way is not evidence for checks 1 or 6.
   Take the host state from git (`git ls-files <path>`, `git log -- <path>`) or from a read the user runs outside the sandbox, and never delete the placeholder.
 
-## The 12 checks
+## The checks
 
 The canonical list lives in
 [docs/setup/secure-agent-setup.md → Verification → Via a Claude Code prompt](../../../../docs/setup/secure-agent-setup.md#via-a-claude-code-prompt-1).
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md b/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md
index 008b438df..f78110213 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/README.md
@@ -10,7 +10,7 @@ Behavioral evals for the `setup-isolated-setup-doctor` skill.
 | Suite | Step | Cases | What it covers |
 |---|---|---|---|
 | `runtime-routing` | Runtime routing | 4 | Codex, Antigravity, Pi and Goose route to their native adapters and never require Claude files |
-| `interpret-probes` | Probe interpretation (`## The 8 probes`) | 19 | all-pass, ssh-fail, localhost-fail, docker-skipped, multiple-fail, ssh-skipped-no-env, injection-in-probe-output, signing-key-fail, gh-sandbox-fail, container-gateway pass/not-running/socket-denied/no-backend/relative-CONTAINER_HOST, scratch-on-shared-session-root, dev-tools-not-granted, git-hooks-unreadable, working-dirs glob entry not matched, working-dirs listing the vetted-ops checkout (✗) |
+| `interpret-probes` | Probe interpretation (`## The probes`) | 19 | all-pass, ssh-fail, localhost-fail, docker-skipped, multiple-fail, ssh-skipped-no-env, injection-in-probe-output, signing-key-fail, gh-sandbox-fail, container-gateway pass/not-running/socket-denied/no-backend/relative-CONTAINER_HOST, scratch-on-shared-session-root, dev-tools-not-granted, git-hooks-unreadable, working-dirs glob entry not matched, working-dirs listing the vetted-ops checkout (✗) |
 | `after-report` | Report synthesis (`## After the report`) | 5 | all-clear-all-pass, all-clear-with-skips, ssh-fail-with-catalog-link, multiple-fail-two-catalog-links, injection-asks-autofix-rejected |
 
 ## Run
diff --git a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json
index bf09b7d8f..3d954f27c 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json
+++ b/tools/skill-evals/evals/setup-isolated-setup-doctor/interpret-probes/fixtures/step-config.json
@@ -1,4 +1,4 @@
 {
   "skill_md": "skills/setup-isolated-setup-doctor/SKILL.md",
-  "step_heading": "## The 8 probes"
+  "step_heading": "## The probes"
 }
diff --git a/tools/skill-evals/evals/setup-isolated-setup-verify/README.md b/tools/skill-evals/evals/setup-isolated-setup-verify/README.md
index 1852ffcd2..38b5312c1 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-verify/README.md
+++ b/tools/skill-evals/evals/setup-isolated-setup-verify/README.md
@@ -11,7 +11,7 @@ Behavioral evals for the `setup-isolated-setup-verify` skill.
 |---|---|---|---|
 | codex-native | Codex security model | 3 | untrusted or absent guard despite project trust, unsafe vetted write allowance, and the credential read-isolation gap |
 | runtime-routing | Runtime routing | 4 | Codex, Antigravity, Pi and Goose route to their native adapters and never require Claude files |
-| step-1-classify | The 12 checks | 24 | all-pass, sandbox disabled, missing scripts, version drift, project root missing, injection attempt, signing key unreadable in the sandbox (check 10), dev-tool paths missing from settings.local.json (check 8 ⚠), whole-user hook dir unreadable from the sandbox (check 8 ✗), working directories missing under the read block with a glob entry (check 15 ⚠), roborev installed with `review` and `show` auto-allowed no config-edit deny and an incomplete `ask` list (check 17b, 17d and 17e ✗), roborev's hook replacing the shared whole-user dispatcher (check 17f ✗), `~/.roborev/` made writable with and without step B.2 (check 17c ✗), a roborev MCP server registered (check 17g ⚠), the vetted-ops checkout listed as a working directory (check 16f ✗), Linux `allowAllUnixSockets` with no host-socket `denyRead` (check 12e ✗), `denyRead` missing from every merged scope (check 1 ✗), settings only in `settings.local.json` beside a sandbox placeholder (check 1 ✓ naming the file), an IDE-launched session with credential-shaped variable names (check 7 ⚠), a plain `ssh-ed25519` signing key with `commit.gpgsign` on (check 10 ✓, the overlay sub-checks n/a), a software signing key with an unrelated smart card plugged in (check 10 ✓, the overlay sub-checks n/a), no sentinel and no `ls` for 7a (check 7 inconclusive), no framework checkout (check 5 ⚠ for the unenforced claude-code floor, checks 10 and 13 unchecked, installed versions still reported) |
+| step-1-classify | The checks | 24 | all-pass, sandbox disabled, missing scripts, version drift, project root missing, injection attempt, signing key unreadable in the sandbox (check 10), dev-tool paths missing from settings.local.json (check 8 ⚠), whole-user hook dir unreadable from the sandbox (check 8 ✗), working directories missing under the read block with a glob entry (check 15 ⚠), roborev installed with `review` and `show` auto-allowed no config-edit deny and an incomplete `ask` list (check 17b, 17d and 17e ✗), roborev's hook replacing the shared whole-user dispatcher (check 17f ✗), `~/.roborev/` made writable with and without step B.2 (check 17c ✗), a roborev MCP server registered (check 17g ⚠), the vetted-ops checkout listed as a working directory (check 16f ✗), Linux `allowAllUnixSockets` with no host-socket `denyRead` (check 12e ✗), `denyRead` missing from every merged scope (check 1 ✗), settings only in `settings.local.json` beside a sandbox placeholder (check 1 ✓ naming the file), an IDE-launched session with credential-shaped variable names (check 7 ⚠), a plain `ssh-ed25519` signing key with `commit.gpgsign` on (check 10 ✓, the overlay sub-checks n/a), a software signing key with an unrelated smart card plugged in (check 10 ✓, the overlay sub-checks n/a), no sentinel and no `ls` for 7a (check 7 inconclusive), no framework checkout (check 5 ⚠ for the unenforced claude-code floor, checks 10 and 13 unchecked, installed versions still reported) |
 | step-2-recommend | After the report | 7 | all-pass, install needed, update needed, project-root missing, multiple gaps, roborev findings (settings edit by the operator, `.roborev.toml` left to them), no framework checkout (warn, with a re-run once a clone exists) |
 
 ## Run
diff --git a/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/step-config.json b/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/step-config.json
index 7e88fc220..e56f02a3e 100644
--- a/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/step-config.json
+++ b/tools/skill-evals/evals/setup-isolated-setup-verify/step-1-classify/fixtures/step-config.json
@@ -1,6 +1,6 @@
 {
   "skill_md": "skills/setup-isolated-setup-verify/SKILL.md",
-  "step_heading": "## The 12 checks",
+  "step_heading": "## The checks",
   "also_include": [
     "skills/setup-isolated-setup-verify/conditional-checks.md"
   ]
diff --git a/tools/spec-loop/specs/sandbox-diagnostics.md b/tools/spec-loop/specs/sandbox-diagnostics.md
index 023b09639..b966b6f6a 100644
--- a/tools/spec-loop/specs/sandbox-diagnostics.md
+++ b/tools/spec-loop/specs/sandbox-diagnostics.md
@@ -74,8 +74,7 @@ error, and two skills that probe or verify the setup on demand.
   exiting 1 so the line reaches the model and the user. Tests under
   `tools/agent-isolation/tests/test_sandbox_error_hint.py`.
 - Skill `setup-isolated-setup-doctor` — live, read-only probes
-  (nine, `### Probe 1` to `### Probe 9`, under a section heading that
-  still reads `## The 8 probes`), each reporting ✓ / ✗ / ⊘ / ⚠ with the command
+  (nine, `### Probe 1` to `### Probe 9`, under `## The probes`), each reporting ✓ / ✗ / ⊘ / ⚠ with the command
   and its output as evidence, and each mapping ✗ to the matching
   catalog anchor. Each probe is a deterministic, side-effect-free
   script, `scripts/probe-<n>-<slug>.sh`, printing one
@@ -113,8 +112,8 @@ error, and two skills that probe or verify the setup on demand.
   which shows what an un-excluded `gh` does on this machine, then
   checks that `"gh *"` is configured.
 - Skill `setup-isolated-setup-verify` — static checks of the installed
-  configuration (numbered 1–17 under a section heading that still reads
-  `## The 12 checks`; 13, 14 and 17 optional): settings shape, hook wiring, hook scripts, wrapper,
+  configuration (numbered 1–17 under `## The checks`; 13, 14 and 17
+  optional): settings shape, hook wiring, hook scripts, wrapper,
   pinned versions, status line, denial canaries, project-root grant
   (check 8 — also the dev-tool paths, whose absence is ⚠ not ✗, and in
   whole-user scope the shared hook directory probed from inside the
@@ -293,8 +292,6 @@ PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner \
   hook branch beside it" rule; the session-start hook's grant cleanup and `isolated-setup-update`'s
   must-fix report are its only other surfaces.
 - Probes 2, 4 and 6–9 still have no script tests.
-- The doctor's and verify skill's section headings (`## The 8 probes`,
-  `## The 12 checks`) lag their bodies (nine probes, seventeen checks).
 - A command that spells a working directory with a literal `~` still
   prompts under the read block even when the directory is listed. For
   the vetted-ops commands the setup closes this by requiring the read