Repository navigation
MIT-652: a turn-nudge signal that needs no tests (eval condition, off) - #181
Merged
Merged
Conversation
…ition COMMONS_TURN_NUDGE=commands nudges when a command failed and later worked in the same turn (a CI check retried until it passed counts), besides a fixed failing test. The hook spools each Bash call's exit code, marks turn ends in the spool, and sends the store one boolean at Stop. Off by default; the shipped texts and `on` are unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
With COMMONS_TURN_NUDGE unset, off or on, the hook keeps no exit codes or turn-end records in the spool, skips the Stop check, and leaves fixed_failing_command out of the Stop body, so the default path collects and sends nothing new (decision 0016). DESIGN.md names the +command-nudge label. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
A turn-nudge signal that needs no tests, built as an eval condition. It is off by default and nothing ships it.
COMMONS_TURN_NUDGE=commands(new setting): the store nudges when the turn fixed a failing test (asondoes) or the Stop hook saw a command fail and later work in the same turn. It has its own text,agent_text.COMMAND_NUDGE, and the prompt-version label+command-nudge.off(the default) andonbehave exactly as before.RELEASED_HOOK_TEXTS, the shipped texts andtest_hook_texts.pyare unchanged.domain/retries.pydefines the signal. The module docstring and the 2026-11-11 update in decision 0034 spell it out:grep,rg,test,[,false,which,command,type,ls,find,diff,cmp,git diff,git grep), since the exit code could be theirs. A CI check exiting 8 (gh pr checks: pending) is not a failure.git commit,npm run). LeadingVAR=valuewords, wrappers (sudo,env,timeout...) and runners (uv run,npx,python -m...) are skipped. An empty subcommand matches any. Shell plumbing (cd,echo,cat,tail,sed...) names no program.gh pr checks, andgh run view|watch --exit-status) count as one program, so any later passing check is a retry.gh run viewwithout--exit-statuskeeps its own name, because it exits 0 whatever the run's result.on.COMMONS_TURN_NUDGE=commands. Under any other setting (the default included), the spool and the Stop body are exactly as before, and a test checks that for unset,offandon:on_tool_usespools each Bash call's exit code.on_stopcomputes the signal over the spool records since the last turn-end marker, writes a new marker, and sendsfixed_failing_command(one boolean) tobackend.stop. No command leaves the machine.LocalBackend,RemoteBackendand the hosted/api/hooks/stoppass the boolean tofinish_turn. Older servers ignore the extra top-level key, as they did whencostwas added.finish_turntakes its hook-side observations as oneTurnSeen(paired,cost,fixed_failing_command). A seventh parameter would have broken ruff's PLR0913 limit, and the grouping keeps that limit as it is.Why
Ticket:
MIT-652(build part).COMMONS_TURN_NUDGEfired only on a test signal. Real misses happened with no local tests, or with a failure that showed only on CI.How it was tested
make checkpassed twice: once in the commit hook, and once after mergingorigin/feature/eval-setup(MIT-682: screen new GitHub tasks for difficulty #177–MIT-681: kill-only same-task verdict, smoke test, store review sheet #179): 1672 passed. MIT-648: propose an admin-only entry flag classifier #180 (one decision doc) was merged afterwards with no conflicts.server/tests/test_command_retries.py:.scratch/eval/github-same10(pip-tools task), cut down to Bash calls and exit codes, with paths scrubbed and output dropped, inserver/tests/recorded_turns/. Each one is checked twice: by the domain function, and replayed through the real hook CLI (post-tool-use/post-tool-use-failure/stop) against Postgres.stop_hook_active) and the next turn don't nudge. An excludedgrepfailure doesn't nudge; a failedgh pr checksand then a passing--watchdoes.offandonignore the signal.test_reliability_gated.py: the store-levelcommandssetting (command signal, test signal, the next turn,onignoring it).test_prompt_versions.py: the+command-nudgelabel and surface.test_hosted_hooks.py: over HTTP, only the boolean crosses and the server nudges.Recorded sessions: which of the 34 saved sessions would have fired (
github-same10andgithub-haiku-none-check, read only, not modified):github-same10-gh-pip-tools-flag-default-3(inlinepython -cfailed, then worked),github-same10-gh-table-extraction-bool-form-13(sed -i ... && pytestfailed, then pytest worked),github-haiku-none-check-gh-mloda-duckdb-join-alias-9andgithub-haiku-none-check-gh-mloda-join-order-5(unpiped pytest failed, then passed)github-haiku-none-check-gh-marklassian-link-labels-7The other 30 don't fire on the command signal. Mostly their failures were hidden behind
| tail(shell exit 0), or the only failure was an excludedgrep(pip-tools-flag-default-10), or a command that failed never worked again (pip-tools-flag-default-9). Those GitHub tasks all have local tests, so they are not the no-test or CI-only misses this targets. That kind of session isn't saved yet.The eval was not run (budget set to none). To run this condition, the multi-turn scenarios plus controls on both models, same shape as
make eval-reliability'sr-nudgeline:Learned
COMMONS_TURN_NUDGEreaches them. ThecommonsMCP server gets onlyENV_KEYSplus the variant key, though, so a nudge condition may not show in the eval's prompt-version label.exit_codeand a{"turn_end": true}record at each Stop._evidenceignores records withoutcommand,textorremoved, sousedis unaffected.vultureflags attribute assignments such asshlex.shlex(...).whitespace_split = True(60% confidence).domain/retries.pyuses a small regex tokenizer instead, which also treats newlines outside quotes as separators, something shlex can't do.tail. The shell's exit code then shows 0, so a signal on raw exit codes rarely sees test failures. Only the runner-summary parsing inobserved_exit_codedoes.🤖 Generated with Claude Code
https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y