Repository navigation
test: pin the answer timeout guard and the eval runner code version - #616
Merged
Merged
Conversation
|
Automatic reviews are disabled for this repository. |
caitlon
force-pushed
the
tests/bcm-309-answer-timeout-guard
branch
from
October 21, 2026 20:09
692aabd to
70b879b
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Problem
The guard in
AssistantService.answer()that lets aTimeoutErrorother than the turn's own deadline propagate had no test: deleting it left the suite green.test_three_failures_in_a_row_stop_the_run_with_exit_3failed now and then. The runner recordsgit describe --dirtyof the live tree as each row's code version, and the test spent a real minute in the pacer between its second and third run. A commit or a first edit in that minute changed the version, so the third run was refused as a provenance change.Change
TimeoutErrorduring generation andanswer()passes it through.git_version, and the three-run test passes--no-pacing, which takes it from about 61 s to under a second.Verification
answer()removed, the new unit test is the only test in its file that fails../scripts/ci/ci-local.sh fast.Migrations
None.
Screenshots
None.