Repository navigation
Conversation
Whisper::generate forced the next token into the timestamp range after
16 tokens without one. The model then treats the forced timestamp as the
end of the segment and often emits EOT, so sentences longer than ~16
tokens are cut mid-word ("...has count <|26.20|>").
On 82 English clips (<= 30 s) through flm serve, same NPU, libs and
weights, this one change takes WER from 7.2% to 3.1%; the drop is almost
all deletions (82 -> 13). Whisper emits timestamps on its own when the
prompt asks for them; nothing needs to force them.
generate() sampled the language from the <|startoftranscript|> logits but never fed it back, so the decoder saw <|startoftranscript|> <|transcribe|>, a prompt Whisper was not trained on. Feed the language token before the task token, as Whisper expects. The translate branch was empty and relied on the loop feeding the language token; it now feeds <|translate|> the same way transcribe feeds <|transcribe|>.
The handler read only model and file, so OpenAI clients that set language or prompt got neither. Forward both to Whisper::generate: - language (ISO 639-1) replaces per-window detection with the given language token; an unknown code is an error, not a silent fallback. - prompt goes before <|startoftranscript|> behind <|startofprev|>, as openai-whisper does: leading space, text tokens only, the last 223 tokens kept. The decode budget shrinks by the prompt length. Both are optional; requests without them behave as before.
The timestamp loop sampled plain argmax with no rules, and its time map missed <|30.00|>: - _build_time_map summed 0.02f while <= 30.00. Float error ends the loop at 29.98, so <|30.00|> (51865) was not a timestamp: never sampled as one, never used to seek. Build the 1501 entries from an integer index. - No pairing rule, so text could follow a closing timestamp, and no limit on the first timestamp, so a window starting mid-speech could open at <|9.80|> and skip what came before. Apply openai-whisper's ApplyTimestampRules: the first timestamp of a window is at most <|1.00|>; after a timestamp that closes a segment only a timestamp or EOT may follow; a segment is not empty; and timestamps never go back. Making <|30.00|> a timestamp without these rules made long audio worse (see the PR), because a window that ends on it then seeks a full 30 s and the next window may skip its start.
For audio over 30 s, generate() moves the next window to the last timestamp it saw. Text emitted after that timestamp was already in the result, and the next window decodes the same audio again, so the tail was duplicated. openai-whisper and faster-whisper drop the unfinished segment before seeking. Hold text after a timestamp in a full window until the next timestamp closes it, and drop it if none does. When a window has no timestamp to resume from, keep the text and skip 30 s as before. Also reset last_time_stamp per window, so a window without timestamps no longer seeks by the previous window's last one. Windows shorter than 30 s, and the no-timestamp mode, stream tokens as before.
/v1/audio/transcriptions printed every transcript to stdout ("Audio
content: ..."). Under a service manager that is the journal, so
dictated speech ends up in system logs. Log it only when
FLM_LOG_RAW_OUTPUT is set, the knob that already re-enables raw model
output for other models.
In a full window, text after the last timestamp is held and dropped, since the next window starts at that timestamp and decodes it again. The timestamp that opened the dropped segment was still emitted, so with return_time_stamp the result had a segment with no text and the same timestamp three times in a row: ...from Linux.<|18.50|><|18.50|><|18.50|> and I saw... Hold the opening timestamp (the second of a pair) with the text, emit both when a timestamp closes the segment, and drop both otherwise. On a 110 s clip the repeated timestamps are gone and the transcript text is unchanged. Co-Authored-By: Claude <[email protected]>
The ASR test's prompt checks failed on every platform: a prompt that repeats
words from the audio ("China's", "He", "Don't") dropped the start of the
transcript. The prompt construction in this PR matches openai-whisper; the
engine was the problem. Rebuilt libwhisper_npu (Linux and Windows, HRX and
XRT) and the five Whisper xclbins from FastFlowLM_IRON caee8bcd, which:
- stops encoder self-attention and decoder cross-attention from attending to
the 36 padding rows after the 1500 audio frames (host side and a key mask
in the encoder attention kernel);
- sets round-to-nearest in the Whisper kernels; they ran with the reset
default, floor, which biased the bfp16-emulated matmuls (encoder fc2 was
10.6% off);
- uses the exact GELU instead of the tanh approximation;
- keeps the encoder and decoder residual stream in fp32, where bf16 rounded
small layer updates away next to outlier channels.
The model weights are unchanged. On the Linux HRX portable build, the first-
word logits now match an fp32 run of the same 4-bit weights, and
asr_test.py --require prompt passes: WER 2.44% with a prompt and without,
names 10/10 (was 41.46% vs 4.88%, 8/10).
Co-Authored-By: Claude <[email protected]>
The server dropped response_format, so every request got {model, text}:
verbose_json had no segments, language or duration, and text, srt and vtt
came back as JSON. The ASR test's verbose_json and timestamp_rules checks
had nothing to run on.
verbose_json, srt and vtt decode with the timestamps returned and split the
output into segments (transcript_format.hpp): text runs from the timestamp
before it to the one after it, or to the end of the clip when a window cut
it off, clamped to the clip. A timestamp can fall inside a multi-byte
character, so those bytes move to the next segment. verbose_json reports
the clip's duration and, as OpenAI, the language in lower case; tokens and
the per-segment statistics are placeholders. text, srt and vtt go out as
text/plain. json and the default are unchanged.
An unknown response_format or language is a 400 invalid_request_error, not
a 500 server_error: the OpenAI SDKs retry a 500.
On a Strix NPU (XRT), asr_test.py with every group required passes all 84
files: verbose_json 8/8, timestamp_rules 2/2, language 13/13.
Co-Authored-By: Claude <[email protected]>
The server now returns verbose_json segments, so a pull request that breaks them, or Whisper's timestamp rules, should fail. Co-Authored-By: Claude <[email protected]>
Collaborator
Author
|
Superseded by #794: both commits (cdc8299 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #794. This branch is #794's head (
f4f9341f) plus two commits; review only those. When #794 merges, this reduces to them. It targetsmainso the ASR test and model sweep run.What
/v1/audio/transcriptionsdroppedresponse_format, so every request got{model, text}:verbose_jsonhad nosegments,languageorduration.text,srtandvttcame back as JSON.verbose_jsonandtimestamp_rulesgroups had nothing to check. Their warning wasserver ignores verbose_json.cdc8299: feat(server): honour response_format
verbose_json,srt,vtt: decode with timestamps returned, then split the output into segments (src/include/whisper/transcript_format.hpp).verbose_jsonfields:durationis the clip length.languageis lowercase, as OpenAI reports it (english).tokens,avg_logprob,compression_ratioandno_speech_probare placeholders that no client filter drops a segment on.text,srt,vttare sent astext/plain; charset=utf-8:send_responsenow writes a bare JSON string as a plain-text body.jsonand the default are unchanged.response_formatorlanguageis now a 400invalid_request_errorinstead of a 500server_error. The OpenAI SDKs retry a 500.asr_test.pynow expects OpenAI's lowercase language names.871ddcf: ci(asr): require verbose_json and timestamp_rules
The workflow comment asks for exactly this once the server passes these groups.
Tested
Strix NPU, Linux XRT, local build of this branch:
asr_test.pywith every group required: all checks pass on the 84 files.verbose_json8/8,timestamp_rules2/2,language13/13 (reported asenglish,japanese, …;xxis rejected with 400),prompt2/2,window_tail4/4,quiet_log1/1.text,srtandvttreturntext/plainbodies; the Japanese SRT is valid UTF-8.response_format=xmlandlanguage=xxreturn HTTP 400.transcript_format.hpp:<|inside speechNot in this PR
timestamp_granularities[]=word, translation (/v1/audio/translations) and streaming.🤖 Generated with Claude Code