Repository navigation
fix(whisper): make the ASR decode loop behave like Whisper - #794
Merged
Merged
Conversation
tawei-amd
pushed a commit
that referenced
this pull request
Oct 16, 2026
The test now measures what PRs #510, #794 and #798 need validated: - assets/asr_eval: 84 FLEURS test files (24.5 min, 6.8 MB) with human transcripts. English clips inside one window, ~83 s and ~308 s files that cross window boundaries, and 10 clips each of de, es, fr, ja, zh and ar. build_asr_eval.py rebuilds it; README.md carries the CC BY 4.0 attribution. Replaces the model-made test_audio reference. - asr_test.py: WER/CER per set and pooled (each file capped at its reference length), runaway transcripts, speed, format agreement, Gemma 4 chat transcription, and feature groups (verbose_json, timestamp_rules, window_tail, language, prompt, quiet_log) that fail the run only when required. --baseline compares with another build's results, --expected with fixed rates. - asr-test.yml: on a pull request, main's latest build runs first on the same runner as the baseline. Pushes and the weekly run compare with expected.json. A dispatch can test any PR's builds and set the required groups and drift limit. window_tail and prompt are required by default, since main passes them. Co-Authored-By: Claude <[email protected]>
8 of 9 tasks
3 of 4 tasks
quine00
pushed a commit
to quine00/FastFlowLM
that referenced
this pull request
Oct 16, 2026
The ASR test failed PR ROCm#798 on Windows HRX with "no more runaways than the baseline: 4 vs 3". The new runaway, de-fleurs-02, loops on its first half in both builds; ROCm#798's build also transcribed the second half that main's dropped. Same 12 errors, more of the real speech, but the extra words pushed it past 1.5 times the reference length. - A file now counts against a build only if it became a runaway, or repeats more phrases, and also has more errors than in the baseline. - The per-set limit applies to sets of ten or more files. A single long-form file moves by many points whenever one decode flips, which HRX builds do between identical runs (main's own Linux HRX build scored 18.5% and 36.8% on it); those files are left to the pooled rate and the two checks above. Replayed against every CI result so far (ROCm#798 and ROCm#799, four platforms each) and local runs of ROCm#510, ROCm#794 and ROCm#798: all pass, including ROCm#798 on Windows HRX. A synthetic new loop with more errors still fails. Co-Authored-By: Claude <[email protected]>
Whisper::generate forced the next token into the timestamp range after
16 tokens without one. The model then treats the forced timestamp as the
end of the segment and often emits EOT, so sentences longer than ~16
tokens are cut mid-word ("...has count <|26.20|>").
On 82 English clips (<= 30 s) through flm serve, same NPU, libs and
weights, this one change takes WER from 7.2% to 3.1%; the drop is almost
all deletions (82 -> 13). Whisper emits timestamps on its own when the
prompt asks for them; nothing needs to force them.
generate() sampled the language from the <|startoftranscript|> logits but never fed it back, so the decoder saw <|startoftranscript|> <|transcribe|>, a prompt Whisper was not trained on. Feed the language token before the task token, as Whisper expects. The translate branch was empty and relied on the loop feeding the language token; it now feeds <|translate|> the same way transcribe feeds <|transcribe|>.
The handler read only model and file, so OpenAI clients that set language or prompt got neither. Forward both to Whisper::generate: - language (ISO 639-1) replaces per-window detection with the given language token; an unknown code is an error, not a silent fallback. - prompt goes before <|startoftranscript|> behind <|startofprev|>, as openai-whisper does: leading space, text tokens only, the last 223 tokens kept. The decode budget shrinks by the prompt length. Both are optional; requests without them behave as before.
The timestamp loop sampled plain argmax with no rules, and its time map missed <|30.00|>: - _build_time_map summed 0.02f while <= 30.00. Float error ends the loop at 29.98, so <|30.00|> (51865) was not a timestamp: never sampled as one, never used to seek. Build the 1501 entries from an integer index. - No pairing rule, so text could follow a closing timestamp, and no limit on the first timestamp, so a window starting mid-speech could open at <|9.80|> and skip what came before. Apply openai-whisper's ApplyTimestampRules: the first timestamp of a window is at most <|1.00|>; after a timestamp that closes a segment only a timestamp or EOT may follow; a segment is not empty; and timestamps never go back. Making <|30.00|> a timestamp without these rules made long audio worse (see the PR), because a window that ends on it then seeks a full 30 s and the next window may skip its start.
For audio over 30 s, generate() moves the next window to the last timestamp it saw. Text emitted after that timestamp was already in the result, and the next window decodes the same audio again, so the tail was duplicated. openai-whisper and faster-whisper drop the unfinished segment before seeking. Hold text after a timestamp in a full window until the next timestamp closes it, and drop it if none does. When a window has no timestamp to resume from, keep the text and skip 30 s as before. Also reset last_time_stamp per window, so a window without timestamps no longer seeks by the previous window's last one. Windows shorter than 30 s, and the no-timestamp mode, stream tokens as before.
/v1/audio/transcriptions printed every transcript to stdout ("Audio
content: ..."). Under a service manager that is the journal, so
dictated speech ends up in system logs. Log it only when
FLM_LOG_RAW_OUTPUT is set, the knob that already re-enables raw model
output for other models.
In a full window, text after the last timestamp is held and dropped, since the next window starts at that timestamp and decodes it again. The timestamp that opened the dropped segment was still emitted, so with return_time_stamp the result had a segment with no text and the same timestamp three times in a row: ...from Linux.<|18.50|><|18.50|><|18.50|> and I saw... Hold the opening timestamp (the second of a pair) with the text, emit both when a timestamp closes the segment, and drop both otherwise. On a 110 s clip the repeated timestamps are gone and the transcript text is unchanged. Co-Authored-By: Claude <[email protected]>
tawei-amd
force-pushed
the
whisper-decode-fixes
branch
from
October 17, 2026 13:40
a1604f2 to
6350e88
Compare
The ASR test's prompt checks failed on every platform: a prompt that repeats
words from the audio ("China's", "He", "Don't") dropped the start of the
transcript. The prompt construction in this PR matches openai-whisper; the
engine was the problem. Rebuilt libwhisper_npu (Linux and Windows, HRX and
XRT) and the five Whisper xclbins from FastFlowLM_IRON caee8bcd, which:
- stops encoder self-attention and decoder cross-attention from attending to
the 36 padding rows after the 1500 audio frames (host side and a key mask
in the encoder attention kernel);
- sets round-to-nearest in the Whisper kernels; they ran with the reset
default, floor, which biased the bfp16-emulated matmuls (encoder fc2 was
10.6% off);
- uses the exact GELU instead of the tanh approximation;
- keeps the encoder and decoder residual stream in fp32, where bf16 rounded
small layer updates away next to outlier channels.
The model weights are unchanged. On the Linux HRX portable build, the first-
word logits now match an fp32 run of the same 4-bit weights, and
asr_test.py --require prompt passes: WER 2.44% with a prompt and without,
names 10/10 (was 41.46% vs 4.88%, 8/10).
Co-Authored-By: Claude <[email protected]>
The server dropped response_format, so every request got {model, text}:
verbose_json had no segments, language or duration, and text, srt and vtt
came back as JSON. The ASR test's verbose_json and timestamp_rules checks
had nothing to run on.
verbose_json, srt and vtt decode with the timestamps returned and split the
output into segments (transcript_format.hpp): text runs from the timestamp
before it to the one after it, or to the end of the clip when a window cut
it off, clamped to the clip. A timestamp can fall inside a multi-byte
character, so those bytes move to the next segment. verbose_json reports
the clip's duration and, as OpenAI, the language in lower case; tokens and
the per-segment statistics are placeholders. text, srt and vtt go out as
text/plain. json and the default are unchanged.
An unknown response_format or language is a 400 invalid_request_error, not
a 500 server_error: the OpenAI SDKs retry a 500.
On a Strix NPU (XRT), asr_test.py with every group required passes all 84
files: verbose_json 8/8, timestamp_rules 2/2, language 13/13.
Co-Authored-By: Claude <[email protected]>
The server now returns verbose_json segments, so a pull request that breaks them, or Whisper's timestamp rules, should fail. Co-Authored-By: Claude <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #793.
Six commits, one per defect. Each builds on its own.
fix(whisper): stop forcing a timestamp every 16 text tokens- the forced timestamp ends segments early and the model then emits EOT. Main cause of the WER gap.fix(whisper): feed the language token to the decoder- the prompt becomes<|startoftranscript|><|lang|><|task|>. The empty translate branch now feeds<|translate|>.feat(server): honour language and prompt in /v1/audio/transcriptions-language(ISO 639-1) replaces detection; an unknown code is an error.promptgoes behind<|startofprev|>as openai-whisper does (leading space, text tokens only, last 223 tokens). Both optional.fix(whisper): apply Whisper's timestamp rules, <|30.00|> included- 1501 timestamps built from an integer index; first timestamp at most<|1.00|>; after a closing timestamp only a timestamp or EOT; no empty segments; timestamps never go back.fix(whisper): do not decode the tail of a full window twice- text after the last timestamp of a full 30 s window is held and dropped, since the next window starts there.last_time_stampis reset per window.fix(server): do not log transcripts by default- "Audio content: ..." is printed only withFLM_LOG_RAW_OUTPUTset.Whisper::generategains two defaulted parameters (language,prompt), soflm runandtest/whisper_npuare unchanged.Validation
Built with
cmake --preset linux-default -DFLM_USE_HRX=ONagainst HRX 2026.10.13, g++ 15.3, Ubuntu 26.10, Krackan (Ryzen AI 7 350), NPU fw 1.1.2.64, model Whisper-V3-Turbo-NPU2 @ 594eecd2. Each commit was built and run throughflm serve, one POST per clip. WER %, CER for ja/zh.languagesetFLEURS, 10 clips per language:
languagesetlanguage(commit 3) fixes it, which is the point of honouring the field.promptworks: on the 302 s clip,prompt="Gryce Challoner Sweetwater"spells both names right (0 -> 2 "Gryce", 0 -> 1 "Challoner"), WER 1.29 -> 1.00.<|30.00|>, greedy decoding ran on past it into a repetition loop, and the next window opened seconds late. That variant is not in this PR.Not run: the Windows build, the XRT build,
flm runwith audio, the CI deb matrix. Compiles clean with the HRX preset on Linux.