Skip to content

feat(hook-manager): add Step P.3b eval suite for hook-manager shim skipping (#1838 follow-up) #1891

Open

Description

@Kaap10

What should happen

Add behavioral eval coverage in tools/skill-evals/evals/setup-isolated-setup-install/ for Step P.3b (Step-P prose) verifying that the agent skips copying shims to ~/.claude/bin/ for hook managers whose install strategy is none (simple-git-hooks, cargo-husky, git-code-format-maven-plugin, git-build-hook) and installs the shim exclusively for managers requiring an install fix (prek).

To ensure incremental review and zero external tool dependencies, this follow-up is structured into two parts:

  • Part 1 (Immediate PR): Implement the step-hook-manager-shim behavioral eval suite in setup-isolated-setup-install covering positive, negative (none-strategy skipping), mixed, and scope boundary cases.
  • Part 2 (Subsequent Follow-Up): Empirical execution of git-code-format-maven-plugin and git-build-hook under the whole-user dispatcher in both main checkout and linked worktrees in an environment with JDK 17+ and Maven 3.9+, allowing documentation to drop the "from upstream source" qualifier.

Why

During the review and merge of PR #1838, maintainer @potiuk noted:

"test_none_strategy_generates_no_shim_case pins that the generated shim has no case for a none adapter. The decision that matters is the install step skipping those adapters, which lives in step-p's prose, so its real coverage would be an eval case for setup-isolated-setup-install. A run of the two Maven plugins under the dispatcher, in the main checkout and a linked worktree, would also let those bullets drop the "from upstream source" qualifier."

The generator test test_none_strategy_generates_no_shim_case pins that tools/agent-isolation/hook-manager-shim.sh contains no case statement for none adapters, but does not test the agent's decision-making during Step P.3b execution. Adding an eval suite pins this behavior against regression.

Which layer

Python package (tools/*/ with pyproject.toml)

Boundary conditions (optional)

  • Negative skipping: Repositories using only strategy: "none" adapters (simple-git-hooks, cargo-husky, git-code-format-maven-plugin, git-build-hook) must produce installed_shims: [].
  • Positive installation: Repositories using prek must produce installed_shims: ["prek"].
  • Mixed repositories: Repositories combining prek with none adapters must selectively install installed_shims: ["prek"].
  • Unmanaged repositories: Repositories without hook managers must produce installed_shims: [].
  • Non-dispatcher scopes: When the operator selects per-project scope, Step P.3b must be skipped entirely (dispatcher_installed: false, installed_shims: []).

Out of scope (optional)

  • Modifying tools/hook-manager/registry.json or existing adapter definitions.
  • Running live Java/Maven execution (Part 2).

References (optional)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions