[tawei-amd]

Summary

Adds an ASR Test workflow that measures speech recognition on real NPU hardware whenever ASR-related code changes. It also gives PRs #510, #794 and #798 a way to be validated on hardware. CI currently has no hardware coverage for ASR, because the model sweep's audio task is disabled.

What it measures (.github/script/asr_test.py):

• Accuracy: WER, or CER for ja and zh, on 84 FLEURS files with human transcripts. Each set and the pooled total are reported, with each file's errors capped at its reference length so that one runaway transcript can't swamp a set.
• Runaways: transcripts longer than 1.5× the reference, which is what a decoder stuck in a repetition loop produces.
• Speed: seconds of processing per second of audio.
• Formats: json, no response_format, and verbose_json must agree.
• Chat transcription: WER of gemma4-it:e2b (whose audio front end E4B and the flash variants share) and gemma4-it:12b (which has its own, modeling_gemma4_12b_audio.cpp).

Feature groups. These only fail the run when required; otherwise they are reported as warnings:

Group	What it checks
verbose_json	Segments are ordered, inside the clip, non-empty, and join into the text; duration equals the clip length
timestamp_rules	Segments never overlap; the first starts within 1 s
window_tail	No phrases repeated across 30 s windows
language	Forcing the language doesn't hurt accuracy; it is reported back; unknown codes are rejected
prompt	A prompt doesn't hurt accuracy, and names in it are spelled as given
quiet_log	No transcripts in the server log

Only prompt is required by default, because it is the only group main passes on every platform. main's Linux HRX build loops on the long-form file, so it fails window_tail; a PR that adds repeats is still caught by the baseline comparison. A PR that makes main pass another group adds it to ASR_REQUIRE in the workflow.

Comparison. On a pull request, main's latest build runs first on the same runner and becomes the baseline. The run fails if:

• the pooled error rate rises by more than 1 point;
• any set rises by more than 5 points (sets have 10 files, so one flipped decode moves them a point or more);
• or the number of runaways grows.

The report also shows how many files changed and the drift from main's transcripts. Pushes to main and the weekly run compare against asr_eval/expected.json instead, with 2 points of slack on the pooled rate (versus 1 point against main) and the same 5 points per set. Its rates are kept per platform, because identical sources decode differently on each: main scores 18.55% pooled on Linux, 22.63% on Linux HRX, 18.61% on Windows and 19.44% on Windows HRX, and some sets differ by up to 5 points. A manual dispatch can test any PR's builds (pr) and set require and max_drift.

Evaluation set (.github/script/assets/asr_eval/, 6.8 MB, CC BY 4.0, attributed in its README):

Set	Files	Audio
en-short	20	single window
en-long	3	~83 s each, two window boundaries
en-longform	1	308 s
de, es, fr, ja, zh, ar	10 each

build_asr_eval.py rebuilds the set deterministically from the FLEURS Parquet files.

Triggers: pull requests and pushes to main that touch any of the following, plus a weekly run and manual dispatch:

• the Whisper engine, its NPU kernels and prebuilt libraries;
• FFmpeg, FFT (fftw, pocketfft) and audio_process_utils;
• the chat-model audio front ends;
• rest_handler.cpp, server.cpp and multipart.*;
• the test's own files.

Linux and Windows are both tested, default and HRX builds alike.

Report. A summary job writes one report to the run's summary page. It starts with a verdict per model and platform. Then each model gets its own section with Linux, Linux HRX, Windows and Windows HRX side by side:

• whisper-v3:turbo: accuracy per set against main; runaways, changed files and speed; checks by group; failures and warnings per platform.
• each chat model: WER against main, plus its own failures.

Each job's own report, results JSON and raw responses are in its artifact. The script also refuses to start if its port is already taken, so it can't end up testing another server by mistake.

Validation of #510, #794 and #798 on hardware

I ran each PR's CI-built Linux portable against main (77bc6bdd) on a Strix NPU. Each run required the default groups plus the ones that PR adds.

	main	#510	#794	#798
Pooled error rate	18.55%	18.55%	7.14%	18.82% (+0.27)
en-short / en-long / longform WER	9.71 / 11.89 / 16.53	same	4.74 / 4.87 / 6.48	9.71 / 11.89 / 15.24
de / es / fr WER	38.1 / 15.4 / 27.7	same	7.0 / 2.9 / 7.0	38.6 / 15.4 / 29.3
ja / zh CER, ar WER	19.4 / 13.5 / 41.7	same	6.8 / 12.4 / 17.7	20.0 / 15.4 / 42.8
Runaways	3	3	0	3
Files changed vs main	–	0	50/84	18/84
Gemma chat WER	5.05%	5.05%	5.05%	5.05%
Result	pass	fail: verbose_json	fail: prompt	pass

#510 (verbose_json). On main's decoder, segments are invalid in 28 of 84 files, 5 of them English short clips:

• they end past the clip (for example 0.00–24.74 s on an 8.6 s clip) in 24 files;
• they are zero-length or empty in 8 files;
• they go backwards in 4 files.

Most of this comes from main's decoder, which #794 fixes. #510 could still clamp segment ends to the duration and drop empty segments.

#794 (decode loop). It reproduces the claimed gains on every set, removes all runaway loops, rejects unknown language codes, and stops logging transcripts. However, prompt looks broken. When a prompt word is spoken in the clip, the transcript drops everything up to it:

• prompt="China's" returns " economic output." instead of the whole sentence;
• the WER of the prompted clips goes from 6.1% to 41.5%.

When the prompt words aren't spoken before the end, it works as intended: "Fichte" is then spelled correctly. The timestamp rules can't be observed without verbose_json (#510).

#798 (pocketfft). No regression: pooled +0.27 points, no new runaways, and Gemma is unchanged. 18 files changed, nearly all by one word or character; one change is a fix (漏れ効いた → 漏れ聞いた). That's the expected effect of slightly different FFT rounding tipping a borderline greedy decode. en-short is byte-identical.

Test plan

• main and the three PR builds run through asr_test.py on a Strix NPU (results above)
• Chat transcription is repeatable at temperature 0 (two runs byte-identical)
• Baseline lookup tested against a fake gh: pull request with an expired main run and a missing HRX baseline, dispatch with a PR number, and push without a baseline
• actionlint is clean apart from the custom runner labels
• Windows and Windows HRX jobs on CI (run 38853168399): main then this PR's build, about 18 min per job; 42/42 checks pass, 0 files changed (this PR changes no engine code)
• main's Linux HRX build measured locally for expected.json: it loops on the long-form file ("the state of the state of …"), 36.8% WER against 16.5% on Linux
• Linux and Linux HRX jobs on CI (run 38865384844): 42/42 and 41/41 checks pass, matching the local runs
• Combined report rendered from the four CI results, plus a failing example (fix(whisper): make the ASR decode loop behave like Whisper #794's prompt checks)
• gemma4-it:12b: first tested by this PR's next CI run. It could not run locally, because the test NPU was busy with another CI job.

🤖 Generated with Claude Code