Repository navigation
fix(ci): stop the ASR test failing on decodes that got no worse - #803
Merged
Merged
Conversation
The ASR test failed PR #798 on Windows HRX with "no more runaways than the baseline: 4 vs 3". The new runaway, de-fleurs-02, loops on its first half in both builds; #798's build also transcribed the second half that main's dropped. Same 12 errors, more of the real speech, but the extra words pushed it past 1.5 times the reference length. - A file now counts against a build only if it became a runaway, or repeats more phrases, and also has more errors than in the baseline. - The per-set limit applies to sets of ten or more files. A single long-form file moves by many points whenever one decode flips, which HRX builds do between identical runs (main's own Linux HRX build scored 18.5% and 36.8% on it); those files are left to the pooled rate and the two checks above. Replayed against every CI result so far (#798 and #799, four platforms each) and local runs of #510, #794 and #798: all pass, including #798 on Windows HRX. A synthetic new loop with more errors still fails. Co-Authored-By: Claude <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The ASR test (#799) failed PR #798 on Windows HRX (run 38939931243) with "no more runaways than the baseline: 4 vs 3". That was a false positive:
de-fleurs-02transcriptmainThe extra correct words pushed #798's transcript past 1.5× the reference length, which is the runaway threshold. The loop itself is
main's decoder bug and occurs in both builds. The flip between them comes from #798's FFT change on a borderline HRX decode.Changes (
.github/script/asr_test.py)main's own Linux HRX build scored 18.5% and 36.8% onen-longform. The long-form sets are still covered by the pooled rate and by the two checks above.Test plan
main: all pass the core checks.es-fleurs-03) still fails, naming the file.🤖 Generated with Claude Code