Skip to content

test: pin the answer timeout guard and the eval runner code version - #616

Merged
caitlon merged 3 commits into
devfrom
tests/bcm-309-answer-timeout-guard
Oct 21, 2026
Merged

caitlon merged 3 commits into
devfrom
tests/bcm-309-answer-timeout-guard

Conversation

@caitlon

@caitlon caitlon commented Oct 21, 2026

Copy link
Copy Markdown
Owner

Problem

The guard in AssistantService.answer() that lets a TimeoutError other than the turn's own deadline propagate had no test: deleting it left the suite green.

test_three_failures_in_a_row_stop_the_run_with_exit_3 failed now and then. The runner records git describe --dirty of the live tree as each row's code version, and the test spent a real minute in the pacer between its second and third run. A commit or a first edit in that minute changed the version, so the third run was refused as a provenance change.

Change

  • A unit test where the answer model raises a foreign TimeoutError during generation and answer() passes it through.
  • The three-run eval test and the test that checks the recorded code version pin git_version, and the three-run test passes --no-pacing, which takes it from about 61 s to under a second.

Verification

  • With the guard in answer() removed, the new unit test is the only test in its file that fails.
  • The flaky failure was reproduced in a scratch clone by editing a file 20 s into the test; it fails at the third run's summary, as the original failure did.
  • ./scripts/ci/ci-local.sh fast.

Migrations

None.

Screenshots

None.

@caitlon caitlon added this to the v1.1-polish milestone Oct 21, 2026
@caitlon caitlon added type:test Test coverage area:api FastAPI backend labels Oct 21, 2026
@greptile-apps

greptile-apps Bot commented Oct 21, 2026

Copy link
Copy Markdown

Automatic reviews are disabled for this repository.

@caitlon caitlon self-assigned this Oct 21, 2026
@github-actions github-actions Bot added the size:S Under 300 changed lines. label Oct 21, 2026
@caitlon
caitlon force-pushed the tests/bcm-309-answer-timeout-guard branch from 692aabd to 70b879b Compare October 21, 2026 20:09
@sonarqubecloud

Copy link
Copy Markdown

@caitlon
caitlon merged commit 9e52f16 into dev Oct 21, 2026
20 checks passed
@caitlon
caitlon deleted the tests/bcm-309-answer-timeout-guard branch October 21, 2026 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:api FastAPI backend size:S Under 300 changed lines. type:test Test coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant