Repository navigation
skill-evals(setup-isolated-setup-verify): bring two fixtures up to the checks - #1881
Merged
Merged
Conversation
…e checks Two step-1-classify cases failed on main because their reports predate checks the skill now makes, so the model's verdict was right and the fixture wrong: - case-7: check 9b has required the personal-layer policy denies since #1787, and check 10a the PermissionDenied and PostToolUseFailure disarm hooks since #1316; the report showed neither, so 9 came out ✗ and 10a ⚠. The report now carries both. - case-23: since #1814 a missing PreToolUse `container-gateway-hook.sh ensure` hook makes 12a ◐; the report now registers it, so 12a-12d pass as the case intends and only 12e fails. Generated-by: Claude Opus 5
potiuk
enabled auto-merge (squash)
October 28, 2026 01:00
github-actions
Bot
added
family:setup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two
setup-isolated-setup-verifystep-1-classifycases fail onmain(noted in #1880). In both, the model's verdict follows the skill. The fixtures are wrong: their reports were written before checks the skill now makes.case-7-signing-key-unreadablehas two problems:Editdenies on the personal-layer policy paths (.git/**/apache-magpie,.svn/apache-magpie,.apache-magpie-local). The report listed only the first three surfaces, so check 9 came out ✗.PermissionDeniedandPostToolUseFailuredisarm hooks. The report had neither, so 10a came out ⚠.The report now carries both, and the expected check 9 evidence names the personal layer.
case-23-linux-all-unix-sockets-exposed: since fix(container-gateway-hook): start the gateway on demand before a container command #1814, a missingPreToolUsecontainer-gateway-hook.sh ensurehook rates 12a ◐. The report now registers that hook, so 12a–12d pass and only 12e fails, which is what the case is meant to test.Evals: both cases pass when re-run on this branch.
Generative AI disclosure: drafted with Claude Code (Claude Opus 5).
🤖 Generated with Claude Code