Skip to content

Optional captions from a faster-whisper container, with speaker turns - #18

Merged
dben merged 3 commits into
mainfrom
ccr-008bba80-xewu4q
Oct 22, 2026
Merged

dben merged 3 commits into
mainfrom
ccr-008bba80-xewu4q

Conversation

@dben

@dben dben commented Oct 22, 2026

Copy link
Copy Markdown
Owner

An optional second container that transcribes episodes, so the private feeds carry captions. pod-block works as before without it: the quick start, the main image and kubectl apply -k deployment/ are unchanged.

The transcription service (whisper/)

  • faster-whisper on the CPU (int8), behind one streaming HTTP endpoint. It works on one episode at a time and stops when the client hangs up.
  • Transcribes in batches of 8 by default (WHISPER_BATCH): about twice as quick on a CPU, and it punctuates better.
  • Optional speaker turns (WHISPER_SPEAKERS=true) via sherpa-onnx: pyannote segmentation and a WeSpeaker voice model. No PyTorch or Hugging Face token; the models (about 35 MB) are fetched on first start. POST /speakers re-runs that step alone with other clustering options.
  • Every package is pinned. faster-whisper 1.2.1 breaks with PyAV 19, so av stays at 18.
  • CI builds and publishes ghcr.io/<repo>-whisper beside the app's image.

In pod-block

  • PODBLOCK_WHISPER_URL turns it on. Each episode cut without a timed transcript from its feed gets a captions job, which sends the uncut download to the service.
  • The words come back with times and are split into captions of at most two 42-character lines. They're saved as WebVTT (rel="captions") among the episode's transcripts, so the existing code shifts them to fit the cut and serves them; a re-cut doesn't need a new transcript.
  • The job names its episode in its payload rather than holding it, so the episode stays ready and in the feeds while it runs. Captions jobs get a worker thread of their own, so transcriptions don't hold up downloads.
  • Speaker turns, when the service sends them, are saved beside the captions. They stay out of the captions and the cuts for now. podblock speakers EPISODE_ID compares them with the episode's repeats: each voice's talk time, its share inside the repeats, voices heard mostly inside the repeats, and lone voices. --threshold and --count re-run only the speaker step, for tuning.

Deployment

  • deployment/whisper/ is a separate kustomization with its own labels, so pod-block's Service never selects its pods. The example uses large-v3-turbo with 6 threads.
  • deployment/README.md has a new "Optional: captions" section covering adding and removing it.

Measured on 4 cores (2.8 GHz Xeon), real episodes

Share of episode length
small, one at a time 0.38
small, batches of 8 0.20
large-v3-turbo, batches of 8 0.43
speakers 0.17

On three real shows, two read their own ads, so speaker turns had nothing to add. On a network show, threshold 0.65 separated the ad voices from the hosts (two voices 67% and 88% inside the repeats), while 0.8 merged most of them into the hosts. The default threshold needs more tuning before anything is cut by it.

Testing

  • The full suite passes (201 tests), and ruff is clean.
  • The service's real HTTP handler is tested with stand-in models, end to end through the captions job.
  • I also ran the real models end to end, and built and ran the image with a read-only root file system as uid 1000.

Generated by Claude Code

An optional transcription service (whisper/: faster-whisper behind one
streaming HTTP endpoint, int8 on the CPU) and a `captions` job that sends
each cut episode's download to it when PODBLOCK_WHISPER_URL is set and the
feed has no timed transcript. What comes back is split into two-line
captions and saved as WebVTT among the episode's transcripts, so it is
shifted to fit the cut and served in the private feeds like a feed's own.

Captions jobs name their episode in the payload rather than holding it, so
the episode stays ready and editable, and they get a worker thread of
their own so transcriptions don't hold up downloads. CI builds the image as
ghcr.io/<repo>-whisper; deployment/whisper/ is its kustomization.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Prc7ntPTFgrdpi1Mv7kfFm
The transcription service now transcribes in batches of 8 by default
(faster-whisper's batched pipeline: about twice as quick on a CPU), and,
with WHISPER_SPEAKERS, finds who speaks when with sherpa-onnx (pyannote
segmentation and a WeSpeaker voice model, on the CPU, no PyTorch or
Hugging Face token). The audio is decoded once for both.

The captions job saves the speaker turns beside the captions; they stay out
of the captions and the cuts for now. `podblock speakers EPISODE_ID` sets
them beside the episode's repeats (each voice's talk, its share inside the
repeats, lone voices, voices mostly inside the repeats), and can ask the
service's new /speakers endpoint again with other clustering options, to
tune them without transcribing again.

The example deployment uses large-v3-turbo with 6 threads.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Prc7ntPTFgrdpi1Mv7kfFm
@dben
dben merged commit 8dee3d8 into main Oct 22, 2026
5 checks passed
@dben
dben deleted the ccr-008bba80-xewu4q branch October 22, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants