Skip to content

MIT-682: screen new GitHub tasks for difficulty - #177

Merged
ZachFara merged 4 commits into
feature/eval-setupfrom
feat/MIT-682
Nov 12, 2026
Merged

ZachFara merged 4 commits into
feature/eval-setupfrom
feat/MIT-682

Conversation

@ZachFara

@ZachFara ZachFara commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

What changed

  • tools/eval/screen.py (new): the difficulty screen as pure functions.
    • screen(runs, reference_median_tokens) -> ScreenResult: PASS if Sonnet's none arm failed at least one run (tests_pass false), or if the runs' median tokens_in + tokens_out is at least SCREEN_TOKEN_FACTOR (2.0) times the reference median. The result names the branch that passed, or the branch numbers behind a FAIL. With no reference (or no runs) it returns REFUSED.
    • reference_median(rows): the median of the counted none-arm, model == "sonnet" lines of github-same10 and github-pilot. It returns None when there are none.
    • record_line / report: what the command prints, including the comment line a passing task records.
  • tools/agent_eval.py: only the subcommand wiring. screen --scenario TASK [--runs 3] takes GitHub tasks only. It computes the reference from counted() ledger lines and refuses (exit 2) before any session starts if there is none. It then runs the none arm on Sonnet through plan_arms and run_jobs under condition github-screen, reads back this run's ledger lines, and prints PASS (exit 0) or FAIL (exit 1) with the numbers. If a session errored or is held as a possible leak, it refuses (exit 2). It also refuses --runs below 1 before any session starts. To stay within main()'s 40-statement limit, the report parser's two statements became one chained call. agent_eval._tokens is gone: task_totals and arm_line now use screen.session_tokens, the one definition.
  • tools/eval/github_tasks.py: a "screened tasks" marker comment after the current ten tasks. Every task added below it must end its comment with the screen's line.
  • docs/eval/README.md: the "GitHub tasks" acceptance criteria now include the screen. A new paragraph covers the rule, the condition, when it refuses, that it runs after check, and the recording format. It also names where the reference comes from: 246,683 tokens over 7 sessions of gh-pip-tools-flag-default (4) and gh-table-extraction-bool-form (3). The 2.0x bar is provisional, and each screened task records the reference median it was decided on, because the median moves as new lines arrive.
  • server/tests/test_eval_screen.py (new): tests for the rule, the command and the comment check.

Why

Every current GitHub task passes with no help on Sonnet and Haiku, so none of them can show what help adds. New tasks must be screened for difficulty before they are accepted (the answer to the Question on task difficulty, item 2(a)).

Ticket: MIT-682

How it was tested

  • server/tests/test_eval_screen.py, 19 tests:
    • The rule: the fail branch, the token branch (at exactly 2.0x), both branches at once, both failing, a missing reference, no runs, and the reference's filtering by condition, arm and model.
    • The command through main(), with runner.run_many and repos.fetch stubbed (no real sessions):
      • It plans three none/sonnet jobs under github-screen and prints PASS with the record line.
      • It exits 1 on FAIL.
      • It refuses before any session when only an invalid reference line exists.
      • It refuses when a session never reached the ledger, when one is held as a possible leak, and for --runs 0 or -1 (with no session started).
    • The comment check: the shipped file has exactly one marker, and exactly the current ten tasks above it. A task appended below it passes with a record_line it printed, and is flagged with no line, with numbers that fail the rule, or with fewer than 3 runs.
  • make check passed in pre-commit on the feature commit, again (1589 passed) after merging feature/eval-setup, and in pre-commit on the fix-round commit, after merging feature/eval-setup at 05873b5.
  • No real session was run (the eval budget is none).

Learned

  • The comment check lives in the test, not in screen.py. Vulture scans only server/src and tools, so a checker used only by tests counts as dead code there. The test's regex is tied to the command's output by building its cases with screen.record_line.
  • agent_eval.main() sits at ruff's 40-statement limit again. The next subcommand needs another merged statement or an options helper.
  • github-screen stays out of verdicts only because compare reads one condition at a time. Anything that pools conditions must leave it out.
  • The current reference covers only two tasks (pip-tools and table-extraction): those were the only none-arm Sonnet sessions of the two smoke conditions.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y

ZachFara and others added 4 commits November 11, 2026 21:18
A new task passes if Sonnet's none arm fails at least one of three runs,
or needs at least twice the median tokens of the smoke-test tasks' none-arm
Sonnet sessions. `agent_eval.py screen` runs the sessions under the
github-screen condition and prints PASS, FAIL or REFUSED; tasks added
below the new marker in github_tasks.py record its line in their comment.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
agent_eval's compare figures now use screen.session_tokens instead of a
copy. The screen command refuses --runs below 1 before any session, and
tests cover a session held as a possible leak. The README names the two
tasks the reference median comes from and calls the 2.0x bar provisional.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
@ZachFara
ZachFara merged commit 0fb62ea into feature/eval-setup Nov 12, 2026
2 checks passed
@ZachFara
ZachFara deleted the feat/MIT-682 branch November 12, 2026 02:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant