Repository navigation
setup evals: ground verify step-2 in the skill's rules; split checkout-behind from snapshot drift in update - #1888
Merged
Merged
Conversation
…rules the skill has Three step-2-recommend cases failed on main: - case-3 expected an update follow-up for snapshot drift, which verify has not handled since #1470 moved it to the pre-flight; the model's "pass, nothing to do" was right. The case keeps its intent through a rule that exists: check 5 ⚠ for a pinned primitive drifted from the framework checkout's pin. - case-5 and case-6 failed on the number of follow-ups: the output spec never said whether two checks with the same remedy share an entry, which the expected answers assumed. It now asks for one entry per skill or command, an operator-only decision folded into the entry for its check, and a hard fail first, which reorders case-5's expected. Generated-by: Claude Opus 5
…napshot drift After the report sent a framework checkout behind origin to `setup upgrade`, described as refreshing the gitignored snapshot. Those are two different things: check 1 prints `git pull --ff-only` for the user's own magpie clone, and `setup upgrade` answers snapshot drift. Give each its own bullet, so a run with both names both follow-ups; the step-after-report case-2 that expected exactly that now passes. Generated-by: Claude Opus 5
github-actions
Bot
added
family:setup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
These are the eval failures that were already failing on
mainwhen #1883 and #1885 were measured. Three were stale fixtures, and one was a real inconsistency in the skill.1.
setup-isolated-setup-verify›step-2-recommend(fixtures only)case-3-update-neededexpected an update follow-up for snapshot drift. Verify stopped handling snapshot drift in refactor(skills): drop the Snapshot drift sections the pre-flight check already covers #1470, which moved it to the pre-flight, so the model's "pass, nothing to do" followed the skill. The case keeps its intent, but now through a rule that exists: check 5 ⚠ forbubblewrapdrifted from the framework checkout's pin, whichremediation.mdmaps tosetup-isolated-setup-update.case-5-multipleandcase-6-roborev-findingsfailed on the number offollow_upentries. The output spec never said whether two checks with the same remedy share an entry, although the expected answers assumed they do. It now says:min_versionfloor) comes first, which reorderscase-5's expected answer2.
setup-isolated-setup-update› "After the report" (skill fix)"Framework checkout behind" pointed at
setup upgrade, described as refreshing the gitignored snapshot. These are two different things:git pull --ff-onlyfor the user's own magpie clonesetup upgradeanswers snapshot drift, per the skill's## Snapshot driftsectionEach now has its own bullet, so a run with both names both follow-ups.
step-after-report/case-2-framework-behindexpected exactly that.Evals
step-2-recommend: 7/7 (was 4/7 onmain)step-after-report: 4/4 (was 3/4)Validator and hooks pass.
Generative AI disclosure: drafted with Claude Code (Claude Opus 5).
🤖 Generated with Claude Code