Repository navigation
MIT-681: kill-only same-task verdict, smoke test, store review sheet - #179
Merged
Merged
Conversation
The same-task set's verdict can now only kill or be not decided, by the fold verdict's own kill rule and constants, and a combined verdict kills if either says kill and continues only on the folds. The ten current GitHub tasks become the smoke test (make eval-smoke, condition github-smoke), which no verdict reads. review-sheet prints a store's pending entries next to their task's upstream fix and issue, linked by the task ids ledger lines now record. The arms record and the eval README are amended to match. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
The smoke-test import had its own new noqa; reuse the existing import line's instead and read the list as github_tasks.SMOKE_TASKS. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
…T-681 # Conflicts: # tools/agent_eval.py
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
comparegives a same-task condition (github-arms-same-task, and each one in the newSAME_TASK_CONDITIONS, which also lists the 2026-11-11 rungithub-same10) only kill or not decided.same_task_verdictmaps the fold verdict's ownverdict()(samePASS_MARGIN,TOKEN_MARGIN,PASS_FLOOR, same kill rule) to kill or not decided. Thegithub-armsfold verdict is unchanged.combined_verdict: kill if the folds or the same-task set say kill, continue only if the folds say continue, inconclusive otherwise.compareprints acombined verdict, grader vN:line under the report of eithergithub-armsorgithub-arms-same-task, once both have counted rows of that grader version.SMOKE_TASKSintools/eval/github_tasks.pynames the ten current tasks.arms --smokeruns them underSMOKE_CONDITION = "github-smoke"and refuses--scenario.make eval-smokecalls it with all four arms,EVAL_SMOKE_RUNS(1) times each.compare --condition github-smokeprintsverdict: none, and the combined verdict never reads it.review-sheet --store NAME. Read only (set_read_only(True), and it refuses a store not on localhost). It prints each pending entry (ascommons pendinglists them) with its id, symptom, cause and fix, next to its task'sfix_urlandissue_url. An entry is linked to its task by itstask_id, through a newtask_idsfield on ledger lines. That field comes fromResult.task_ids=Activity.ids: the session's hook tasks plus every task id it passed to a tool, which grading already gathered. An entry with no matching line printstask unknown.fix_urlusingreview-sheet, step 4 combined verdict).docs/eval/README.mdgains "The store review" and "The smoke test" sections, plus thetask_idsledger field, and its compare and run sections are updated to match.Why
Ticket: MIT-681. The ten GitHub tasks were too easy and had no siblings, so the folds were empty and only the inflated same-task set had results. Also, on the table-extraction task, 4 of 6 Commons sessions copied an approved entry's workaround fix. The owner chose kill-only for the same-task set, the ten tasks as a smoke test, and a review step that compares each entry against the upstream fix.
How it was tested
make checkpassed, run by the commit hook on the tree merged with the latestfeature/eval-setup: 1586 tests at the first run. The first run's only failure was vulture flaggingconn.read_only = True, which becameconn.set_read_only(True).server/tests/test_eval_compare.pyuse hand-made ledgers only. One ledger kills and one would continue but prints "not decided", under both same-task conditions. There is a unit test of the kill rule on fixedComparisons, the combined rule's truth table, and the combined line printed under both conditions for three fold/same-task pairs. A smoke test checks that smoke rows (a clear win on the folds' own task names) leave the fold, same-task and combined verdicts as they were, and thatgithub-smokeprintsverdict: none.server/tests/test_eval_store.pyuse the existing store fixture on the test server. They cover:review-sheetwith a linked entry, an unlinked one, a reviewed one that is left out, and a ledger line withouttask_ids, with the entry count unchanged afterwards. They also cover a graded seed-wave session whosetask_idsreach its ledger line and link its entry,review-sheetrefusing a missing or non-local store, andarms --smokeplanning 10 tasks x 4 arms undergithub-smoke.Learned
task_ids, soreview-sheetshowstask unknownfor every entry in stores seeded before it (including thegithub-same10-seedstore). Linking those entries needs a new seed wave, or a transcript-based lookup: each saved<stem>.jsonlholds the Claudesession_id, which matchestasks.session_idin the store.conn.read_only = Trueas an unused attribute. Useconn.set_read_only(True).main()intools/agent_eval.pyis at ruff's 40-statement limit (PLR0915). A new subcommand needs a helper, asarms_report_commands(compare and review-sheet) andclear_optionsdo.git merge --ff-only), so a single hook run covers the merged tree.🤖 Generated with Claude Code
https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y