Skip to content

bench: measure the optional DeepFilterNet denoiser for real (roadmap row 5) - #62

Merged
RevolutionLA merged 2 commits into
mainfrom
feat/df-real-benchmark
Oct 11, 2026
Merged

RevolutionLA merged 2 commits into
mainfrom
feat/df-real-benchmark

Conversation

@RevolutionLA

Copy link
Copy Markdown
Owner

What this answers

Roadmap row 5: "deep-filter cost per utterance is unmeasured" — finish line
"one 10 s utterance with the real binary, wall-clock and RTF recorded, or the
option is retired"
.

bench_asr --denoise drives the add-on's own AudioCapture::dfDenoise(), so
the 16k→48k resample, the per-utterance posix_spawn, the 5 s bounded wait and
the speexdsp fallback are inside the measurement rather than around it.
BACKENDS="deepfilter" tools/benchmark/run_bench.sh DIR produces the row.

The number (i7-1260P, powersave, deep_filter 0.5.6)

backend ok RTF (warm) wall median peak RSS
deepfilter 7/7 0.156 1223 ms 0.06 GB

Fit: wall_ms ≈ 282 + 0.110 × audio_ms (R² 0.89) over 4.69–10.05 s clips; a
repeat run moved RTF by 3% (0.151). Machine header + raw JSONL are committed in
docs/benchmarks/, and the binary is pinned by sha256 in the doc.

Verdict: the option stays. ~1.2 s per utterance lands between key-release
and first character, and the 5 s deadline only binds past ~40 s of continuous
speech — past that the take falls back instead of hanging. speexdsp (default)
costs single-digit ms, and the READMEs now say so.

What the measurement found on the way

#61 — the same path drops
69–97 ms off the end of every take, and most of it is ours: both soxr_process()
legs are called once with no drain, so the resampler's filter-delay tail is
discarded (isolated with a length-preserving stub: 40–64 ms lost), plus ~30 ms
that deep-filter's own framing takes. Because denoise runs before
trimSilence() and that trim only keeps 20 ms of trailing audio, this eats into
the protection the README promises for the last syllable — hence the caveat in
README §5. Not fixed here; that is a separate, audio-behaviour change.

Harness contract (new test, 21st)

A benchmark that lies is worse than none, so tests/test_bench_denoise.sh pins
against stub binaries (the 36 MB real binary can't be a CI dependency):

  • a working stub → ok:true, changed:true, and audio back at ~16 kHz (a 48 kHz
    leak would be 3× this length);
  • a stub that exits 0 without touching the file → ok:false and an error
    field, because cer.py counts a row as measured unless it carries one — without
    this the published table would read 7/7 over seven dead rows;
  • after three consecutive failures the remaining files say not measured and
    carry no wall_ms, so a dead denoiser can never print a timing number;
  • missing binary → failure, with the "not installed" line on stderr.

All three failure-direction assertions were mutation-checked (drop the error
field → red; disable the breaker → red; report samples.size()*3 → red).
meson test 21/21.

Drive-by fixes in scope

  • docs/benchmarks.md linked its artifacts as docs/<file> while they live in
    docs/benchmarks/ — every "prove the number" link 404'd.
  • run_bench.sh wrote the real hostname into the committed machine header (the
    previous artifact was redacted by hand). It is now (redacted) unless
    HOST_LABEL is set, and the header records which deep-filter answered.
  • README/§5: "~100 MB model plus a Rust binary" → the x86_64 musl asset is one
    36 MB static binary with the model compiled in (verified: deep_filter 0.5.6,
    36 417 296 B).

Notes

  • subdir('tools') now precedes subdir('tests') so the new test can pass
    bench_asr.full_path(); no other ordering dependency exists between them.
  • Local machine: ~/.local/share/nextless/bin/deep-filter now holds the measured
    binary. The user's audio.json still says speexdsp, so nothing in the live
    input path changed.

…row 5)

Row 5 asked whether one-shot deep-filter is affordable per utterance, because
every test so far drove it with a stub binary. bench_asr --denoise now calls the
add-on's own AudioCapture::dfDenoise(), so the resample, the per-utterance
spawn, the 5 s deadline and the fallback are all inside the number: 7/7
utterances of 4.7-10 s, median 1223 ms, RTF 0.156, 60 MB child RSS on an
i7-1260P with deep_filter 0.5.6 (sha256 pinned in the doc). The option stays,
and both READMEs now state what it costs.

The measurement also found issue #61: the same path drops 69-97 ms off the end
of every take, most of it from our own unflushed soxr round trip rather than the
model.

tests/test_bench_denoise.sh pins the harness contract against stubs (the real
36 MB binary cannot be a CI dependency): a silently no-oping denoiser must be
reported as a failure AND carry an error field, since the published table counts
a row as measured unless it does; a row past three consecutive failures carries
no timing number; and audio must come back at 16 kHz, not 48 kHz. All three
were mutation-checked. Suite is 21/21.

Also here: run_bench.sh stops baking the hostname into the committed machine
header (HOST_LABEL opts in - the previous artifact was redacted by hand, which
is a step someone forgets), and the artifact links in docs/benchmarks.md now
point at docs/benchmarks/, where those files actually live.
bench_asr returns 2 for "not available on this box" and 3 for "nothing was
denoised"; run_bench.sh treats both as legitimate answers, so write that down in
the file header rather than leaving it to be read off the code.

The README snippet only worked for a binary already on PATH. The release asset is
a single static binary you download and symlink yourself, so say which case each
line is for - both READMEs.
@RevolutionLA
RevolutionLA merged commit faf4947 into main Oct 11, 2026
8 checks passed
@RevolutionLA
RevolutionLA deleted the feat/df-real-benchmark branch October 11, 2026 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant