Skip to content

MIT-652: a turn-nudge signal that needs no tests (eval condition, off) - #181

Merged
ZachFara merged 5 commits into
feature/eval-setupfrom
feat/MIT-652
Nov 12, 2026
Merged

ZachFara merged 5 commits into
feature/eval-setupfrom
feat/MIT-652

Conversation

@ZachFara

@ZachFara ZachFara commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

What changed

A turn-nudge signal that needs no tests, built as an eval condition. It is off by default and nothing ships it.

  • COMMONS_TURN_NUDGE=commands (new setting): the store nudges when the turn fixed a failing test (as on does) or the Stop hook saw a command fail and later work in the same turn. It has its own text, agent_text.COMMAND_NUDGE, and the prompt-version label +command-nudge. off (the default) and on behave exactly as before. RELEASED_HOOK_TEXTS, the shipped texts and test_hook_texts.py are unchanged.
  • domain/retries.py defines the signal. The module docstring and the 2026-11-11 update in decision 0034 spell it out:
    • Failing: a known non-zero exit code, from a run that names a program and has none of the commands that fail with nothing wrong (grep, rg, test, [, false, which, command, type, ls, find, diff, cmp, git diff, git grep), since the exit code could be theirs. A CI check exiting 8 (gh pr checks: pending) is not a failure.
    • Worked afterwards: a later run exits 0 and runs the same program and subcommand (git commit, npm run). Leading VAR=value words, wrappers (sudo, env, timeout...) and runners (uv run, npx, python -m...) are skipped. An empty subcommand matches any. Shell plumbing (cd, echo, cat, tail, sed...) names no program.
    • CI checks (gh pr checks, and gh run view|watch --exit-status) count as one program, so any later passing check is a retry. gh run view without --exit-status keeps its own name, because it exits 0 whatever the run's result.
    • At most one nudge per turn. A turn that searched or proposed gets none, as with on.
  • Hook path, reusing the existing turn-nudge path. All of it happens only when the hook's own environment has COMMONS_TURN_NUDGE=commands. Under any other setting (the default included), the spool and the Stop body are exactly as before, and a test checks that for unset, off and on:
    • on_tool_use spools each Bash call's exit code.
    • on_stop computes the signal over the spool records since the last turn-end marker, writes a new marker, and sends fixed_failing_command (one boolean) to backend.stop. No command leaves the machine.
    • LocalBackend, RemoteBackend and the hosted /api/hooks/stop pass the boolean to finish_turn. Older servers ignore the extra top-level key, as they did when cost was added.
  • finish_turn takes its hook-side observations as one TurnSeen (paired, cost, fixed_failing_command). A seventh parameter would have broken ruff's PLR0913 limit, and the grouping keeps that limit as it is.

Why

Ticket: MIT-652 (build part). COMMONS_TURN_NUDGE fired only on a test signal. Real misses happened with no local tests, or with a failure that showed only on CI.

How it was tested

  • make check passed twice: once in the commit hook, and once after merging origin/feature/eval-setup (MIT-682: screen new GitHub tasks for difficulty #177–MIT-681: kill-only same-task verdict, smoke test, store review sheet #179): 1672 passed. MIT-648: propose an admin-only entry flag classifier #180 (one decision doc) was merged afterwards with no conflicts.
  • server/tests/test_command_retries.py:
    • Unit cases: how programs are named, a different subcommand doesn't match, order matters and unknown exit codes don't count, each excluded command, plumbing, a CI retry, and a pending check or log view that isn't a retry.
    • Recorded transcripts: three sessions from .scratch/eval/github-same10 (pip-tools task), cut down to Bash calls and exit codes, with paths scrubbed and output dropped, in server/tests/recorded_turns/. Each one is checked twice: by the domain function, and replayed through the real hook CLI (post-tool-use / post-tool-use-failure / stop) against Postgres.
    • Hand-made cases through the hooks: two failures in one turn nudge once, and the follow-up Stop (stop_hook_active) and the next turn don't nudge. An excluded grep failure doesn't nudge; a failed gh pr checks and then a passing --watch does. off and on ignore the signal.
  • test_reliability_gated.py: the store-level commands setting (command signal, test signal, the next turn, on ignoring it).
  • test_prompt_versions.py: the +command-nudge label and surface.
  • test_hosted_hooks.py: over HTTP, only the boolean crosses and the server nudges.

Recorded sessions: which of the 34 saved sessions would have fired (github-same10 and github-haiku-none-check, read only, not modified):

Signal Fires Sessions
Command signal (new) 4 of 34 github-same10-gh-pip-tools-flag-default-3 (inline python -c failed, then worked), github-same10-gh-table-extraction-bool-form-13 (sed -i ... && pytest failed, then pytest worked), github-haiku-none-check-gh-mloda-duckdb-join-alias-9 and github-haiku-none-check-gh-mloda-join-order-5 (unpiped pytest failed, then passed)
Test signal (runner summaries) 33 of 34 all but github-haiku-none-check-gh-marklassian-link-labels-7

The other 30 don't fire on the command signal. Mostly their failures were hidden behind | tail (shell exit 0), or the only failure was an excluded grep (pip-tools-flag-default-10), or a command that failed never worked again (pip-tools-flag-default-9). Those GitHub tasks all have local tests, so they are not the no-test or CI-only misses this targets. That kind of session isn't saved yet.

The eval was not run (budget set to none). To run this condition, the multi-turn scenarios plus controls on both models, same shape as make eval-reliability's r-nudge line:

for model in sonnet claude-opus-5-5; do
  COMMONS_TURN_NUDGE=commands uv run python tools/agent_eval.py run \
    --scenario multi-turn-rename-then-default --scenario multi-turn-retry-then-timezone \
    --scenario control-typo --scenario control-documented-flag --runs 3 \
    --model $model --condition r-command-nudge-$model;
done

Learned

  • The eval runner gives the session's hooks the full environment, so COMMONS_TURN_NUDGE reaches them. The commons MCP server gets only ENV_KEYS plus the variant key, though, so a nudge condition may not show in the eval's prompt-version label.
  • The hook spool now holds Bash calls' exit_code and a {"turn_end": true} record at each Stop. _evidence ignores records without command, text or removed, so used is unaffected.
  • vulture flags attribute assignments such as shlex.shlex(...).whitespace_split = True (60% confidence). domain/retries.py uses a small regex tokenizer instead, which also treats newlines outside quotes as separators, something shlex can't do.
  • On the saved GitHub-task sessions, agents pipe almost every test run into tail. The shell's exit code then shows 0, so a signal on raw exit codes rarely sees test failures. Only the runner-summary parsing in observed_exit_code does.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y

ZachFara and others added 5 commits November 11, 2026 21:46
…ition

COMMONS_TURN_NUDGE=commands nudges when a command failed and later worked
in the same turn (a CI check retried until it passed counts), besides a
fixed failing test. The hook spools each Bash call's exit code, marks turn
ends in the spool, and sends the store one boolean at Stop. Off by
default; the shipped texts and `on` are unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
With COMMONS_TURN_NUDGE unset, off or on, the hook keeps no exit codes or
turn-end records in the spool, skips the Stop check, and leaves
fixed_failing_command out of the Stop body, so the default path collects
and sends nothing new (decision 0016). DESIGN.md names the +command-nudge
label.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
@ZachFara
ZachFara merged commit 1119d61 into feature/eval-setup Nov 12, 2026
2 checks passed
@ZachFara
ZachFara deleted the feat/MIT-652 branch November 12, 2026 02:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant