Skip to content

fix(ci): stop the ASR test failing on decodes that got no worse - #803

Merged
tawei-amd merged 1 commit into
mainfrom
fix/asr-test-false-positives
Oct 16, 2026
Merged

tawei-amd merged 1 commit into
mainfrom
fix/asr-test-false-positives

Conversation

@tawei-amd

Copy link
Copy Markdown
Collaborator

Summary

The ASR test (#799) failed PR #798 on Windows HRX (run 38939931243) with "no more runaways than the baseline: 4 vs 3". That was a false positive:

Build de-fleurs-02 transcript Errors
main the first half repeated twice, the second half missing 12
#798 the same repeated first half, plus the correct second half 12

The extra correct words pushed #798's transcript past 1.5× the reference length, which is the runaway threshold. The loop itself is main's decoder bug and occurs in both builds. The flip between them comes from #798's FFT change on a borderline HRX decode.

Changes (.github/script/asr_test.py)

  • Runaways and repeated phrases. A file only counts against a build if it became a runaway (or repeats more phrases) and also has more errors than in the baseline. The check is renamed "no new runaway transcripts" and names the files when it fails.
  • Per-set limit only for sets of ten or more files. A single long-form file moves by many points whenever one decode flips, and HRX builds flip between identical runs: main's own Linux HRX build scored 18.5% and 36.8% on en-longform. The long-form sets are still covered by the pooled rate and by the two checks above.

Test plan

🤖 Generated with Claude Code

The ASR test failed PR #798 on Windows HRX with "no more runaways than
the baseline: 4 vs 3". The new runaway, de-fleurs-02, loops on its
first half in both builds; #798's build also transcribed the second
half that main's dropped. Same 12 errors, more of the real speech, but
the extra words pushed it past 1.5 times the reference length.

- A file now counts against a build only if it became a runaway, or
  repeats more phrases, and also has more errors than in the baseline.
- The per-set limit applies to sets of ten or more files. A single
  long-form file moves by many points whenever one decode flips, which
  HRX builds do between identical runs (main's own Linux HRX build
  scored 18.5% and 36.8% on it); those files are left to the pooled
  rate and the two checks above.

Replayed against every CI result so far (#798 and #799, four platforms
each) and local runs of #510, #794 and #798: all pass, including #798
on Windows HRX. A synthetic new loop with more errors still fails.

Co-Authored-By: Claude <[email protected]>
@tawei-amd
tawei-amd merged commit a50bb53 into main Oct 16, 2026
22 of 35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant