diff --git a/tools/skill-evals/README.md b/tools/skill-evals/README.md index e3390bf25..5e5ec2bb7 100644 --- a/tools/skill-evals/README.md +++ b/tools/skill-evals/README.md @@ -12,7 +12,7 @@ Behavioral eval harness for Apache Magpie skills. Each eval suite tests a skill Suites are currently implemented for: - **setup** — 152 cases across 33 steps (step-verify-drift, step-overrides-surface, step-override-bypass, step-m3-baseline-pick, step-m4-install-gates, step-m5-no-repo-offer, step-m5-companion-offer, step-upgrade-companion-offer, step-adopt-settings-merge, verify-default-set, uninstall-default-set, lock-marketplace-parse, adopt-write-floor, setup-prefill-from-floor, preflight-floor, upgrade-adoption-split, verify-floor, adopt-review-process, adopt-security-list, step-reconcile, step-verify, step-config-stamp, step-config-adversarial, step-config-mail, step-adopt-stamp, step-upgrade-stamp, step-delegate-propose, config-vcs-backend, config-local-custom-parser, adopt-isolation-warning, step-vetted-ops-no-policy, step-unknown-subaction) -- **setup-isolated-setup-install** — 21 cases across 5 steps (runtime-routing, step-snapshot-drift, step-scope-confirm, step-hardware-key, step-roborev-exclusion) +- **setup-isolated-setup-install** — 28 cases across 6 steps (runtime-routing, step-snapshot-drift, step-scope-confirm, step-hardware-key, step-roborev-exclusion, step-hook-manager-shim) - **setup-privacy-llm** — 6 cases across 2 steps (step-1-resolve, step-4-gate) - **setup-shared-config-sync** — 12 cases across 2 steps (step-3-decide-action, step-5-draft-commit) - **pairing-multi-agent-review** — 15 cases across 6 steps (step-1-collect-diff, step-2a-correctness-pass, step-2b-security-pass, step-2c-conventions-pass, step-3-merge-findings, step-4-compose-report) diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/README.md b/tools/skill-evals/evals/setup-isolated-setup-install/README.md index ed62cddd7..8bc51690c 100644 --- a/tools/skill-evals/evals/setup-isolated-setup-install/README.md +++ b/tools/skill-evals/evals/setup-isolated-setup-install/README.md @@ -5,7 +5,7 @@ Behavioral evals for the `setup-isolated-setup-install` skill. -## Suites (21 cases total) +## Suites (28 cases total) | Suite | Step | Cases | What it covers | |---|---|---|---| @@ -14,6 +14,7 @@ Behavioral evals for the `setup-isolated-setup-install` skill. | step-scope-confirm | #### Step P.0 — scope choice | 4 | per-project fresh, whole-user with disclosure, settings.json conflict → diff-and-ask, injection resistance | | step-hardware-key | ### Step K — Hardware security key (optional) | 4 | operator declines → skipped; `gpg.format=ssh` with `aut` already `Cached` → no slot change, since `aut` is the signing slot (plus the `git config` lines that point git's own signing program and ssh command at the overlay's `wrap` mode); OpenPGP signing with `aut` `Cached` → `aut off` proposed so ssh transport stops asking for a touch; injection in pasted `ykman` output resisted, `sig` → `cached` only | | step-roborev-exclusion | ## Step B — The roborev exclusion (optional) | 5 | roborev on `PATH` → the exclusion with the catch-all `ask` and both config `Edit` denies, no `allow`; roborev absent → skipped; the operator asking for an unprompted `review` → still no `allow`; macOS operator asking for the hook with read+write → B.2 with `allowRead` only; Linux operator asking for the hook → no B.2 (`allowUnixSockets` is ignored there) | +| step-hook-manager-shim | ### Step P.3b-whole-user | 7 | whole-user dispatcher shim installation; installs shims only for adapters with install.strategy other than none that a repo the operator works in uses (`prek` today); skips strategy: none managers (`simple-git-hooks`, `cargo-husky`, `git-build-hook`, `git-code-format-maven-plugin`) | ## Run @@ -47,3 +48,4 @@ uv run --project tools/skill-evals skill-eval \ flag it, propose `cached` for the operator to run, and never `fixed`. - `case-3-settings-conflict` must set `proceed: false` — the skill stops and diffs the existing file before writing, as required by the golden rules. +- `step-hook-manager-shim` cases are fully auto-comparable: `dispatcher_installed` is boolean and `installed_shims` is a string array. diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-1-prek-repo/case-meta.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-1-prek-repo/case-meta.json new file mode 100644 index 000000000..b09bc04b5 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-1-prek-repo/case-meta.json @@ -0,0 +1 @@ +{"tags":["smoke","local-smoke"]} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-1-prek-repo/expected.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-1-prek-repo/expected.json new file mode 100644 index 000000000..6a642f481 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-1-prek-repo/expected.json @@ -0,0 +1 @@ +{"dispatcher_installed": true, "installed_shims": ["prek"]} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-1-prek-repo/report.md b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-1-prek-repo/report.md new file mode 100644 index 000000000..f0a34d3fa --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-1-prek-repo/report.md @@ -0,0 +1,6 @@ + + +Operator selection: whole-user dispatcher flavour. +Repositories the operator works in: + - /home/dev/project-a: contains prek.toml at root diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-2-simple-git-hooks-repo/case-meta.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-2-simple-git-hooks-repo/case-meta.json new file mode 100644 index 000000000..03128a3ee --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-2-simple-git-hooks-repo/case-meta.json @@ -0,0 +1 @@ +{"tags":["local-smoke","smoke"]} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-2-simple-git-hooks-repo/expected.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-2-simple-git-hooks-repo/expected.json new file mode 100644 index 000000000..0e8f82b4f --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-2-simple-git-hooks-repo/expected.json @@ -0,0 +1 @@ +{"dispatcher_installed": true, "installed_shims": []} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-2-simple-git-hooks-repo/report.md b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-2-simple-git-hooks-repo/report.md new file mode 100644 index 000000000..9e3564284 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-2-simple-git-hooks-repo/report.md @@ -0,0 +1,6 @@ + + +Operator selection: whole-user dispatcher flavour. +Repositories the operator works in: + - /home/dev/frontend-repo: contains package.json with "simple-git-hooks" configuration diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-3-cargo-husky-repo/case-meta.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-3-cargo-husky-repo/case-meta.json new file mode 100644 index 000000000..dd6be101a --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-3-cargo-husky-repo/case-meta.json @@ -0,0 +1 @@ +{"tags":["smoke"]} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-3-cargo-husky-repo/expected.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-3-cargo-husky-repo/expected.json new file mode 100644 index 000000000..0e8f82b4f --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-3-cargo-husky-repo/expected.json @@ -0,0 +1 @@ +{"dispatcher_installed": true, "installed_shims": []} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-3-cargo-husky-repo/report.md b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-3-cargo-husky-repo/report.md new file mode 100644 index 000000000..9554fd761 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-3-cargo-husky-repo/report.md @@ -0,0 +1,6 @@ + + +Operator selection: whole-user dispatcher flavour. +Repositories the operator works in: + - /home/dev/rust-service: Rust project with installed hook marker in .git/hooks/pre-push (cargo-husky) diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-4-maven-repo/case-meta.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-4-maven-repo/case-meta.json new file mode 100644 index 000000000..dd6be101a --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-4-maven-repo/case-meta.json @@ -0,0 +1 @@ +{"tags":["smoke"]} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-4-maven-repo/expected.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-4-maven-repo/expected.json new file mode 100644 index 000000000..0e8f82b4f --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-4-maven-repo/expected.json @@ -0,0 +1 @@ +{"dispatcher_installed": true, "installed_shims": []} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-4-maven-repo/report.md b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-4-maven-repo/report.md new file mode 100644 index 000000000..49d92799c --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-4-maven-repo/report.md @@ -0,0 +1,6 @@ + + +Operator selection: whole-user dispatcher flavour. +Repositories the operator works in: + - /home/dev/backend-java: contains pom.xml configuring com.cosium.code:git-code-format-maven-plugin and git-build-hook diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-5-mixed-repos/case-meta.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-5-mixed-repos/case-meta.json new file mode 100644 index 000000000..b09bc04b5 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-5-mixed-repos/case-meta.json @@ -0,0 +1 @@ +{"tags":["smoke","local-smoke"]} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-5-mixed-repos/expected.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-5-mixed-repos/expected.json new file mode 100644 index 000000000..6a642f481 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-5-mixed-repos/expected.json @@ -0,0 +1 @@ +{"dispatcher_installed": true, "installed_shims": ["prek"]} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-5-mixed-repos/report.md b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-5-mixed-repos/report.md new file mode 100644 index 000000000..d2c7e6c83 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-5-mixed-repos/report.md @@ -0,0 +1,8 @@ + + +Operator selection: whole-user dispatcher flavour. +Repositories the operator works in: + - /home/dev/core: contains prek.toml at root + - /home/dev/web: contains package.json with "simple-git-hooks" + - /home/dev/app: contains pom.xml with git-code-format-maven-plugin diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-6-unmanaged-repo/case-meta.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-6-unmanaged-repo/case-meta.json new file mode 100644 index 000000000..dd6be101a --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-6-unmanaged-repo/case-meta.json @@ -0,0 +1 @@ +{"tags":["smoke"]} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-6-unmanaged-repo/expected.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-6-unmanaged-repo/expected.json new file mode 100644 index 000000000..0e8f82b4f --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-6-unmanaged-repo/expected.json @@ -0,0 +1 @@ +{"dispatcher_installed": true, "installed_shims": []} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-6-unmanaged-repo/report.md b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-6-unmanaged-repo/report.md new file mode 100644 index 000000000..95c15457c --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-6-unmanaged-repo/report.md @@ -0,0 +1,6 @@ + + +Operator selection: whole-user dispatcher flavour. +Repositories the operator works in: + - /home/dev/docs-repo: standard git repository with Markdown files only; no hook manager configs or hook markers diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-7-non-dispatcher-scope/case-meta.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-7-non-dispatcher-scope/case-meta.json new file mode 100644 index 000000000..b09bc04b5 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-7-non-dispatcher-scope/case-meta.json @@ -0,0 +1 @@ +{"tags":["smoke","local-smoke"]} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-7-non-dispatcher-scope/expected.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-7-non-dispatcher-scope/expected.json new file mode 100644 index 000000000..25a70a95a --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-7-non-dispatcher-scope/expected.json @@ -0,0 +1 @@ +{"dispatcher_installed": false, "installed_shims": []} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-7-non-dispatcher-scope/report.md b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-7-non-dispatcher-scope/report.md new file mode 100644 index 000000000..6e975ae17 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/case-7-non-dispatcher-scope/report.md @@ -0,0 +1,6 @@ + + +Operator selection: per-project scope (Step P.0). +Repositories the operator works in: + - /home/dev/project-a: contains prek.toml at root diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/output-spec.md b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/output-spec.md new file mode 100644 index 000000000..075af11a0 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/output-spec.md @@ -0,0 +1,17 @@ + + +## Output format + +Return ONLY valid JSON with this structure: + +```json +{ + "dispatcher_installed": true | false, + "installed_shims": ["", ...] +} +``` + +- `dispatcher_installed`: whether the proposal installs `~/.claude/git-hooks/git-hook-dispatcher.sh` and hook symlinks. +- `installed_shims`: lists every hook-manager binary copied to `~/.claude/bin/`. When multiple shims are installed, list them sorted alphabetically. +- Do not include any text outside the JSON object. diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/step-config.json b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/step-config.json new file mode 100644 index 000000000..949842223 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/step-config.json @@ -0,0 +1,7 @@ +{ + "skill_md": "skills/setup-isolated-setup-install/step-p-sandbox-allowlists.md", + "step_heading": "### Step P.3b-whole-user — Install the per-repo dispatcher + hook-manager shim (dispatcher flavour only)", + "also_include": [ + "tools/hook-manager/registry.json" + ] +} diff --git a/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/user-prompt-template.md b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/user-prompt-template.md new file mode 100644 index 000000000..c72928685 --- /dev/null +++ b/tools/skill-evals/evals/setup-isolated-setup-install/step-hook-manager-shim/fixtures/user-prompt-template.md @@ -0,0 +1,8 @@ + + +## Operator hook-manager environment and repositories + +{report} + +Determine whether the dispatcher and hook-manager PATH shims should be installed, and return JSON only.