Skip to content

MIT-681: kill-only same-task verdict, smoke test, store review sheet - #179

Merged
ZachFara merged 4 commits into
feature/eval-setupfrom
feat/MIT-681
Nov 12, 2026
Merged

ZachFara merged 4 commits into
feature/eval-setupfrom
feat/MIT-681

Conversation

@ZachFara

Copy link
Copy Markdown
Owner

What changed

  • Same-task verdict is kill-only. compare gives a same-task condition (github-arms-same-task, and each one in the new SAME_TASK_CONDITIONS, which also lists the 2026-11-11 run github-same10) only kill or not decided. same_task_verdict maps the fold verdict's own verdict() (same PASS_MARGIN, TOKEN_MARGIN, PASS_FLOOR, same kill rule) to kill or not decided. The github-arms fold verdict is unchanged.
  • Combined rule. combined_verdict: kill if the folds or the same-task set say kill, continue only if the folds say continue, inconclusive otherwise. compare prints a combined verdict, grader vN: line under the report of either github-arms or github-arms-same-task, once both have counted rows of that grader version.
  • Smoke test. SMOKE_TASKS in tools/eval/github_tasks.py names the ten current tasks. arms --smoke runs them under SMOKE_CONDITION = "github-smoke" and refuses --scenario. make eval-smoke calls it with all four arms, EVAL_SMOKE_RUNS (1) times each. compare --condition github-smoke prints verdict: none, and the combined verdict never reads it.
  • review-sheet --store NAME. Read only (set_read_only(True), and it refuses a store not on localhost). It prints each pending entry (as commons pending lists them) with its id, symptom, cause and fix, next to its task's fix_url and issue_url. An entry is linked to its task by its task_id, through a new task_ids field on ledger lines. That field comes from Result.task_ids = Activity.ids: the session's hook tasks plus every task id it passed to a tool, which grading already gathered. An entry with no matching line prints task unknown.
  • Docs. The arms record has a dated note giving the three reasons, and amends items 3 and 9 and the run sequence (step 0 smoke, step 2 review against fix_url using review-sheet, step 4 combined verdict). docs/eval/README.md gains "The store review" and "The smoke test" sections, plus the task_ids ledger field, and its compare and run sections are updated to match.

Why

Ticket: MIT-681. The ten GitHub tasks were too easy and had no siblings, so the folds were empty and only the inflated same-task set had results. Also, on the table-extraction task, 4 of 6 Commons sessions copied an approved entry's workaround fix. The owner chose kill-only for the same-task set, the ten tasks as a smoke test, and a review step that compares each entry against the upstream fix.

How it was tested

  • make check passed, run by the commit hook on the tree merged with the latest feature/eval-setup: 1586 tests at the first run. The first run's only failure was vulture flagging conn.read_only = True, which became conn.set_read_only(True).
  • New tests in server/tests/test_eval_compare.py use hand-made ledgers only. One ledger kills and one would continue but prints "not decided", under both same-task conditions. There is a unit test of the kill rule on fixed Comparisons, the combined rule's truth table, and the combined line printed under both conditions for three fold/same-task pairs. A smoke test checks that smoke rows (a clear win on the folds' own task names) leave the fold, same-task and combined verdicts as they were, and that github-smoke prints verdict: none.
  • New tests in server/tests/test_eval_store.py use the existing store fixture on the test server. They cover: review-sheet with a linked entry, an unlinked one, a reviewed one that is left out, and a ledger line without task_ids, with the entry count unchanged afterwards. They also cover a graded seed-wave session whose task_ids reach its ledger line and link its entry, review-sheet refusing a missing or non-local store, and arms --smoke planning 10 tasks x 4 arms under github-smoke.
  • No eval sessions were run.

Learned

  • Ledger lines from before this change have no task_ids, so review-sheet shows task unknown for every entry in stores seeded before it (including the github-same10-seed store). Linking those entries needs a new seed wave, or a transcript-based lookup: each saved <stem>.jsonl holds the Claude session_id, which matches tasks.session_id in the store.
  • vulture flags conn.read_only = True as an unused attribute. Use conn.set_read_only(True).
  • main() in tools/agent_eval.py is at ruff's 40-statement limit (PLR0915). A new subcommand needs a helper, as arms_report_commands (compare and review-sheet) and clear_options do.
  • Each commit runs the full check through pre-commit. Under a limit on check runs, fast-forward onto the working branch before the commit when the incoming files don't overlap yours (git merge --ff-only), so a single hook run covers the merged tree.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y

ZachFara and others added 4 commits November 11, 2026 21:26
The same-task set's verdict can now only kill or be not decided, by the
fold verdict's own kill rule and constants, and a combined verdict kills
if either says kill and continues only on the folds. The ten current
GitHub tasks become the smoke test (make eval-smoke, condition
github-smoke), which no verdict reads. review-sheet prints a store's
pending entries next to their task's upstream fix and issue, linked by
the task ids ledger lines now record. The arms record and the eval
README are amended to match.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
The smoke-test import had its own new noqa; reuse the existing import
line's instead and read the list as github_tasks.SMOKE_TASKS.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
@ZachFara
ZachFara merged commit 85e167e into feature/eval-setup Nov 12, 2026
2 checks passed
@ZachFara
ZachFara deleted the feat/MIT-681 branch November 12, 2026 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant