Repository navigation
fix(security): the fix loop cannot game its own checker, and can give up - #72
Merged
Merged
Conversation
A. Verifier integrity. applyDiff refuses, with PROTECTED, any fix diff (model or recipe) that touches tests, test runner config or the test script, Penthera's own files (policy, suppressions, lock, audit/baseline.json, .penthera/), CI workflows, git hooks, lockfiles or .gitmodules. One list, next to the old .gitmodules check, on canonical segments. A model diff that does this is reported as refused, not silently dropped. The loop also refuses to verify a fix if any protected file changed since the run began, checked before the tests run and again before the verdict. B. Safe exit. "Could not fix" is recorded as a gave_up event with a reason (verification failed, diff refused, or no usable model fix). The finding is not retried in that run, gave_up never touches recipe success, and the run summary lists what was given up and why. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Actions A and B from
docs/research/agent-incidents-2026.md. This is hardening during the 1.8.0 freeze: please don't merge yet. The maintainer decides whether it goes into 1.8.0 or 1.9. No version bump.A. Verifier integrity
applyDiff(lib/fix/apply.js, one listPROTECTED_FILESnext to the old.gitmodulescheck) refuses any fix diff, from a local model or from a recipe, that touches the files below. The error isPROTECTEDand tells the user to make that change by hand. Matching runs on canonical segments, so case changes, trailing dots and 8.3 aliases are refused too.test/,tests/,__tests__/,*.test.*,*.spec.*,conftest.pyvitest.*,jest.*,playwright.*,pytest.ini,pyproject.toml,setup.cfg,tox.ini,package.json(holdsscripts.testand jest config)penthera.*(policy, suppress, lock),audit/baseline.json,.penthera/.github/workflows/.husky/,.githooks/,lefthook.yml,.pre-commit-config.yaml(.git/was already refused).gitmodules, Windows 8.3 short names (X~1)How verification is protected: the brief offered two ways. I picked the simpler one, refuse instead of restore. Fix diffs can no longer edit a protected file, and fix commits use
--onlywith the fix's own files, so HEAD's judge always matches the base. At the start of the run the loop records every protected file that differs frombaseSha(tracked diffs plus untracked files, with their hashes). It checks this snapshot again before the tests run and before the verdict. If anything changed, the fix fails and the tests are not run. Comparing against the start of the run, not against base alone, keeps--allow-dirtyusable when the user already has their own edits to a test file.When a model diff edits a protected file,
model.plannow throwsPROTECTEDinstead of returningnull.planFixesreports this asrefused, sofixprints it and the loop records it. Before, it was dropped silently, which also makes E3's tamper rate measurable.B. Safe exit
gave_up({ id, reason, source, loop }), records three cases: the fix failed verification, its diff was refused (PROTECTED, dirty file, …), or--fix-modelgave no usable fix.buildViewsignoresgave_up, so recipe success and--auto lownever count it. A recipe's own verification failure still writesfix_failedas before. Only the give-up is neutral.loopRun.gaveUpholds[{ fid, title, reason }], and the CLI printsgave up on N (not retried this run; fix by hand): title (reason); ….Code: about 95 lines across
lib/fix/{apply,index,model}.js,lib/memory/store.js,src/agent/loop.jsandsrc/cli/{agent,fix}.js.Tests
tests/fix.test.js› verifier integrity: a fix never edits its own checkerrefuses every protected path with PROTECTED and says to make the change by hand(29 paths, including case variants and 8.3 aliases)still lets a fix edit ordinary source files with similar namesrefuses a model diff that edits a test file to make it pass, and reports itholds a recipe's diff to the same list (defence in depth): a recipe that setsscripts.testtoexit 0tests/agent.test.js› agent loop: verifier integrity and the safe exitrefuses a model diff that edits a test file to make it pass, gives the finding up and records gave_updoes not retry a finding after one honest failure in a runrecords a model that gives no usable fix as gave_up, oncenever counts a give-up as a recipe failure in memorydoes not verify a fix when tests changed during the run, and never runs themprints what it gave up and why in the run summaryTo check that the tests catch regressions, I turned off the judge snapshot and the give-up filter one at a time. The matching tests failed each time.
npm test(1021 passed),npm run lintandnpm run audit:internal(every gate passed) all pass locally.Notes for review
gave_upevent type, so they mark a memory chain that contains one as malformed. They then fail closed: no ranking, no--auto. This is noted in the CHANGELOG.package.jsonandpyproject.tomlare protected as a whole, because each holds the test script or runner config. No recipe edits them today. Model fixes were already told not to add dependencies.docs/commands/fix.md,docs/local-loop.md, and the Security section of CHANGELOG [Unreleased].🤖 Generated with Claude Code