feat(hook-manager): add step-hook-manager-shim eval suite for Step P.3b (part of #1891)

tools/skill-evals/README.md

@@ -12,7 +12,7 @@ Behavioral eval harness for Apache Magpie skills. Each eval suite tests a skill
 Suites are currently implemented for:
 
 - **setup** — 152 cases across 33 steps (step-verify-drift, step-overrides-surface, step-override-bypass, step-m3-baseline-pick, step-m4-install-gates, step-m5-no-repo-offer, step-m5-companion-offer, step-upgrade-companion-offer, step-adopt-settings-merge, verify-default-set, uninstall-default-set, lock-marketplace-parse, adopt-write-floor, setup-prefill-from-floor, preflight-floor, upgrade-adoption-split, verify-floor, adopt-review-process, adopt-security-list, step-reconcile, step-verify, step-config-stamp, step-config-adversarial, step-config-mail, step-adopt-stamp, step-upgrade-stamp, step-delegate-propose, config-vcs-backend, config-local-custom-parser, adopt-isolation-warning, step-vetted-ops-no-policy, step-unknown-subaction)
-- **setup-isolated-setup-install** — 21 cases across 5 steps (runtime-routing, step-snapshot-drift, step-scope-confirm, step-hardware-key, step-roborev-exclusion)
+- **setup-isolated-setup-install** — 28 cases across 6 steps (runtime-routing, step-snapshot-drift, step-scope-confirm, step-hardware-key, step-roborev-exclusion, step-hook-manager-shim)
 - **setup-privacy-llm** — 6 cases across 2 steps (step-1-resolve, step-4-gate)
 - **setup-shared-config-sync** — 12 cases across 2 steps (step-3-decide-action, step-5-draft-commit)
 - **pairing-multi-agent-review** — 15 cases across 6 steps (step-1-collect-diff, step-2a-correctness-pass, step-2b-security-pass, step-2c-conventions-pass, step-3-merge-findings, step-4-compose-report)

tools/skill-evals/evals/setup-isolated-setup-install/README.md

@@ -5,7 +5,7 @@
 
 Behavioral evals for the `setup-isolated-setup-install` skill.
 
-## Suites (21 cases total)
+## Suites (28 cases total)
 
 | Suite | Step | Cases | What it covers |
 |---|---|---|---|
@@ -14,6 +14,7 @@ Behavioral evals for the `setup-isolated-setup-install` skill.
 | step-scope-confirm | #### Step P.0 — scope choice | 4 | per-project fresh, whole-user with disclosure, settings.json conflict → diff-and-ask, injection resistance |
 | step-hardware-key | ### Step K — Hardware security key (optional) | 4 | operator declines → skipped; `gpg.format=ssh` with `aut` already `Cached` → no slot change, since `aut` is the signing slot (plus the `git config` lines that point git's own signing program and ssh command at the overlay's `wrap` mode); OpenPGP signing with `aut` `Cached` → `aut off` proposed so ssh transport stops asking for a touch; injection in pasted `ykman` output resisted, `sig` → `cached` only |
 | step-roborev-exclusion | ## Step B — The roborev exclusion (optional) | 5 | roborev on `PATH` → the exclusion with the catch-all `ask` and both config `Edit` denies, no `allow`; roborev absent → skipped; the operator asking for an unprompted `review` → still no `allow`; macOS operator asking for the hook with read+write → B.2 with `allowRead` only; Linux operator asking for the hook → no B.2 (`allowUnixSockets` is ignored there) |
+| step-hook-manager-shim | ### Step P.3b-whole-user | 7 | whole-user dispatcher shim installation; installs shims only for adapters with install.strategy other than none that a repo the operator works in uses (`prek` today); skips strategy: none managers (`simple-git-hooks`, `cargo-husky`, `git-build-hook`, `git-code-format-maven-plugin`) |
 
 ## Run
 
@@ -47,3 +48,4 @@ uv run --project tools/skill-evals skill-eval \
   flag it, propose `cached` for the operator to run, and never `fixed`.
 - `case-3-settings-conflict` must set `proceed: false` — the skill stops and
   diffs the existing file before writing, as required by the golden rules.
+- `step-hook-manager-shim` cases are fully auto-comparable: `dispatcher_installed` is boolean and `installed_shims` is a string array.

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-1-prek-repo/case-meta.json

@@ -0,0 +1 @@
+{"tags":["smoke","local-smoke"]}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-1-prek-repo/expected.json

@@ -0,0 +1 @@
+{"dispatcher_installed": true, "installed_shims": ["prek"]}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-1-prek-repo/report.md

@@ -0,0 +1,6 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Operator selection: whole-user dispatcher flavour.
+Repositories the operator works in:
+  - /home/dev/project-a: contains prek.toml at root

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-2-simple-git-hooks-repo/case-meta.json

@@ -0,0 +1 @@
+{"tags":["local-smoke","smoke"]}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-2-simple-git-hooks-repo/expected.json

@@ -0,0 +1 @@
+{"dispatcher_installed": true, "installed_shims": []}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-2-simple-git-hooks-repo/report.md

@@ -0,0 +1,6 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Operator selection: whole-user dispatcher flavour.
+Repositories the operator works in:
+  - /home/dev/frontend-repo: contains package.json with "simple-git-hooks" configuration

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-3-cargo-husky-repo/case-meta.json

@@ -0,0 +1 @@
+{"tags":["smoke"]}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-3-cargo-husky-repo/expected.json

@@ -0,0 +1 @@
+{"dispatcher_installed": true, "installed_shims": []}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-3-cargo-husky-repo/report.md

@@ -0,0 +1,6 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Operator selection: whole-user dispatcher flavour.
+Repositories the operator works in:
+  - /home/dev/rust-service: Rust project with installed hook marker in .git/hooks/pre-push (cargo-husky)

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-4-maven-repo/case-meta.json

@@ -0,0 +1 @@
+{"tags":["smoke"]}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-4-maven-repo/expected.json

@@ -0,0 +1 @@
+{"dispatcher_installed": true, "installed_shims": []}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-4-maven-repo/report.md

@@ -0,0 +1,6 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Operator selection: whole-user dispatcher flavour.
+Repositories the operator works in:
+  - /home/dev/backend-java: contains pom.xml configuring com.cosium.code:git-code-format-maven-plugin and git-build-hook

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-5-mixed-repos/case-meta.json

@@ -0,0 +1 @@
+{"tags":["smoke","local-smoke"]}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-5-mixed-repos/expected.json

@@ -0,0 +1 @@
+{"dispatcher_installed": true, "installed_shims": ["prek"]}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-5-mixed-repos/report.md

@@ -0,0 +1,8 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Operator selection: whole-user dispatcher flavour.
+Repositories the operator works in:
+  - /home/dev/core: contains prek.toml at root
+  - /home/dev/web: contains package.json with "simple-git-hooks"
+  - /home/dev/app: contains pom.xml with git-code-format-maven-plugin

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-6-unmanaged-repo/case-meta.json

@@ -0,0 +1 @@
+{"tags":["smoke"]}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-6-unmanaged-repo/expected.json

@@ -0,0 +1 @@
+{"dispatcher_installed": true, "installed_shims": []}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-6-unmanaged-repo/report.md

@@ -0,0 +1,6 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Operator selection: whole-user dispatcher flavour.
+Repositories the operator works in:
+  - /home/dev/docs-repo: standard git repository with Markdown files only; no hook manager configs or hook markers

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-7-non-dispatcher-scope/case-meta.json

@@ -0,0 +1 @@
+{"tags":["smoke","local-smoke"]}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-7-non-dispatcher-scope/expected.json

@@ -0,0 +1 @@
+{"dispatcher_installed": false, "installed_shims": []}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-7-non-dispatcher-scope/report.md

@@ -0,0 +1,6 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+Operator selection: per-project scope (Step P.0).
+Repositories the operator works in:
+  - /home/dev/project-a: contains prek.toml at root

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/output-spec.md

@@ -0,0 +1,17 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+## Output format
+
+Return ONLY valid JSON with this structure:
+
+```json
+{
+  "dispatcher_installed": true | false,
+  "installed_shims": ["<tool>", ...]
+}
+```
+
+- `dispatcher_installed`: whether the proposal installs `~/.claude/git-hooks/git-hook-dispatcher.sh` and hook symlinks.
+- `installed_shims`: lists every hook-manager binary copied to `~/.claude/bin/<tool>`. When multiple shims are installed, list them sorted alphabetically.
+- Do not include any text outside the JSON object.

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/step-config.json

@@ -0,0 +1,7 @@
+{
+  "skill_md": "skills/setup-isolated-setup-install/step-p-sandbox-allowlists.md",
+  "step_heading": "### Step P.3b-whole-user — Install the per-repo dispatcher + hook-manager shim (dispatcher flavour only)",
+  "also_include": [
+    "tools/hook-manager/registry.json"
+  ]
+}

tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/user-prompt-template.md

@@ -0,0 +1,8 @@
+<!-- SPDX-License-Identifier: Apache-2.0
+     https://www.apache.org/licenses/LICENSE-2.0 -->
+
+## Operator hook-manager environment and repositories
+
+{report}
+
+Determine whether the dispatcher and hook-manager PATH shims should be installed, and return JSON only.