Repository navigation
MIT-682: screen new GitHub tasks for difficulty - #177
Merged
Merged
Conversation
A new task passes if Sonnet's none arm fails at least one of three runs, or needs at least twice the median tokens of the smoke-test tasks' none-arm Sonnet sessions. `agent_eval.py screen` runs the sessions under the github-screen condition and prints PASS, FAIL or REFUSED; tasks added below the new marker in github_tasks.py record its line in their comment. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
agent_eval's compare figures now use screen.session_tokens instead of a copy. The screen command refuses --runs below 1 before any session, and tests cover a session held as a possible leak. The README names the two tasks the reference median comes from and calls the 2.0x bar provisional. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
tools/eval/screen.py(new): the difficulty screen as pure functions.screen(runs, reference_median_tokens) -> ScreenResult: PASS if Sonnet'snonearm failed at least one run (tests_passfalse), or if the runs' mediantokens_in + tokens_outis at leastSCREEN_TOKEN_FACTOR(2.0) times the reference median. The result names the branch that passed, or the branch numbers behind a FAIL. With no reference (or no runs) it returns REFUSED.reference_median(rows): the median of the countednone-arm,model == "sonnet"lines ofgithub-same10andgithub-pilot. It returns None when there are none.record_line/report: what the command prints, including the comment line a passing task records.tools/agent_eval.py: only the subcommand wiring.screen --scenario TASK [--runs 3]takes GitHub tasks only. It computes the reference fromcounted()ledger lines and refuses (exit 2) before any session starts if there is none. It then runs thenonearm on Sonnet throughplan_armsandrun_jobsunder conditiongithub-screen, reads back this run's ledger lines, and prints PASS (exit 0) or FAIL (exit 1) with the numbers. If a session errored or is held as a possible leak, it refuses (exit 2). It also refuses--runsbelow 1 before any session starts. To stay withinmain()'s 40-statement limit, thereportparser's two statements became one chained call.agent_eval._tokensis gone:task_totalsandarm_linenow usescreen.session_tokens, the one definition.tools/eval/github_tasks.py: a "screened tasks" marker comment after the current ten tasks. Every task added below it must end its comment with the screen's line.docs/eval/README.md: the "GitHub tasks" acceptance criteria now include the screen. A new paragraph covers the rule, the condition, when it refuses, that it runs aftercheck, and the recording format. It also names where the reference comes from: 246,683 tokens over 7 sessions ofgh-pip-tools-flag-default(4) andgh-table-extraction-bool-form(3). The 2.0x bar is provisional, and each screened task records the reference median it was decided on, because the median moves as new lines arrive.server/tests/test_eval_screen.py(new): tests for the rule, the command and the comment check.Why
Every current GitHub task passes with no help on Sonnet and Haiku, so none of them can show what help adds. New tasks must be screened for difficulty before they are accepted (the answer to the Question on task difficulty, item 2(a)).
Ticket: MIT-682
How it was tested
server/tests/test_eval_screen.py, 19 tests:main(), withrunner.run_manyandrepos.fetchstubbed (no real sessions):none/sonnetjobs undergithub-screenand prints PASS with the record line.invalidreference line exists.--runs 0or-1(with no session started).record_lineit printed, and is flagged with no line, with numbers that fail the rule, or with fewer than 3 runs.make checkpassed in pre-commit on the feature commit, again (1589 passed) after mergingfeature/eval-setup, and in pre-commit on the fix-round commit, after mergingfeature/eval-setupat 05873b5.Learned
screen.py. Vulture scans onlyserver/srcandtools, so a checker used only by tests counts as dead code there. The test's regex is tied to the command's output by building its cases withscreen.record_line.agent_eval.main()sits at ruff's 40-statement limit again. The next subcommand needs another merged statement or an options helper.github-screenstays out of verdicts only becausecomparereads one condition at a time. Anything that pools conditions must leave it out.none-arm Sonnet sessions of the two smoke conditions.🤖 Generated with Claude Code
https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y