Repository navigation
bench: measure the optional DeepFilterNet denoiser for real (roadmap row 5) - #62
Merged
Merged
Conversation
…row 5) Row 5 asked whether one-shot deep-filter is affordable per utterance, because every test so far drove it with a stub binary. bench_asr --denoise now calls the add-on's own AudioCapture::dfDenoise(), so the resample, the per-utterance spawn, the 5 s deadline and the fallback are all inside the number: 7/7 utterances of 4.7-10 s, median 1223 ms, RTF 0.156, 60 MB child RSS on an i7-1260P with deep_filter 0.5.6 (sha256 pinned in the doc). The option stays, and both READMEs now state what it costs. The measurement also found issue #61: the same path drops 69-97 ms off the end of every take, most of it from our own unflushed soxr round trip rather than the model. tests/test_bench_denoise.sh pins the harness contract against stubs (the real 36 MB binary cannot be a CI dependency): a silently no-oping denoiser must be reported as a failure AND carry an error field, since the published table counts a row as measured unless it does; a row past three consecutive failures carries no timing number; and audio must come back at 16 kHz, not 48 kHz. All three were mutation-checked. Suite is 21/21. Also here: run_bench.sh stops baking the hostname into the committed machine header (HOST_LABEL opts in - the previous artifact was redacted by hand, which is a step someone forgets), and the artifact links in docs/benchmarks.md now point at docs/benchmarks/, where those files actually live.
bench_asr returns 2 for "not available on this box" and 3 for "nothing was denoised"; run_bench.sh treats both as legitimate answers, so write that down in the file header rather than leaving it to be read off the code. The README snippet only worked for a binary already on PATH. The release asset is a single static binary you download and symlink yourself, so say which case each line is for - both READMEs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this answers
Roadmap row 5: "
deep-filtercost per utterance is unmeasured" — finish line"one 10 s utterance with the real binary, wall-clock and RTF recorded, or the
option is retired".
bench_asr --denoisedrives the add-on's ownAudioCapture::dfDenoise(), sothe 16k→48k resample, the per-utterance
posix_spawn, the 5 s bounded wait andthe speexdsp fallback are inside the measurement rather than around it.
BACKENDS="deepfilter" tools/benchmark/run_bench.sh DIRproduces the row.The number (i7-1260P, powersave,
deep_filter 0.5.6)Fit:
wall_ms ≈ 282 + 0.110 × audio_ms(R² 0.89) over 4.69–10.05 s clips; arepeat run moved RTF by 3% (0.151). Machine header + raw JSONL are committed in
docs/benchmarks/, and the binary is pinned by sha256 in the doc.Verdict: the option stays. ~1.2 s per utterance lands between key-release
and first character, and the 5 s deadline only binds past ~40 s of continuous
speech — past that the take falls back instead of hanging. speexdsp (default)
costs single-digit ms, and the READMEs now say so.
What the measurement found on the way
#61 — the same path drops
69–97 ms off the end of every take, and most of it is ours: both
soxr_process()legs are called once with no drain, so the resampler's filter-delay tail is
discarded (isolated with a length-preserving stub: 40–64 ms lost), plus ~30 ms
that deep-filter's own framing takes. Because denoise runs before
trimSilence()and that trim only keeps 20 ms of trailing audio, this eats intothe protection the README promises for the last syllable — hence the caveat in
README §5. Not fixed here; that is a separate, audio-behaviour change.
Harness contract (new test, 21st)
A benchmark that lies is worse than none, so
tests/test_bench_denoise.shpinsagainst stub binaries (the 36 MB real binary can't be a CI dependency):
ok:true,changed:true, and audio back at ~16 kHz (a 48 kHzleak would be 3× this length);
ok:falseand anerrorfield, because
cer.pycounts a row as measured unless it carries one — withoutthis the published table would read
7/7over seven dead rows;not measuredandcarry no
wall_ms, so a dead denoiser can never print a timing number;All three failure-direction assertions were mutation-checked (drop the error
field → red; disable the breaker → red; report
samples.size()*3→ red).meson test21/21.Drive-by fixes in scope
docs/benchmarks.mdlinked its artifacts asdocs/<file>while they live indocs/benchmarks/— every "prove the number" link 404'd.run_bench.shwrote the real hostname into the committed machine header (theprevious artifact was redacted by hand). It is now
(redacted)unlessHOST_LABELis set, and the header records whichdeep-filteranswered.36 MB static binary with the model compiled in (verified:
deep_filter 0.5.6,36 417 296 B).
Notes
subdir('tools')now precedessubdir('tests')so the new test can passbench_asr.full_path(); no other ordering dependency exists between them.~/.local/share/nextless/bin/deep-filternow holds the measuredbinary. The user's
audio.jsonstill saysspeexdsp, so nothing in the liveinput path changed.