[dben]

Two commits: the transcription service can now run as several replicas across nodes, and every job now records how long each step takes, with an admin page to read the results.

Several transcription replicas

• pod-block: PODBLOCK_CAPTION_WORKERS sets how many captions jobs run at once. Set it to the number of whisper replicas.
• Finding a free replica: a busy replica answers {"waiting": true} as soon as a request arrives. With more than one captions worker, pod-block hangs up and sends the episode again after a backoff (2 s, doubling up to 30 s), so the Service can pick another replica. After 10 minutes of busy answers it waits its turn where it is.
• Cut-off transcriptions (a replica restarting, say) are asked for once more before the job fails.
• Reading the answer: lines are now read as they arrive, with urllib3's read1. iter_lines() held them back until 512 bytes had come, so a short "waiting" line could go unseen for minutes.
• whisper server, stopping: on SIGTERM it finishes the episode in hand, sends away the requests waiting, answers 503 to new ones, then exits. Before this, the server ignored SIGTERM (it runs as PID 1) and was killed after the grace period.
• whisper server, shared models: replicas share one models folder. The first to start holds /models/.fetch.lock while it downloads; the others wait, then load what it fetched. Speaker models are downloaded under temporary names and moved into place whole.
• Manifests:
  • two replicas on a ReadWriteMany models volume (storageClassName marked CHANGE, with a note on what to do without one);
  • rolling updates that start the new pod before stopping the old one;
  • terminationGracePeriodSeconds: 3600;
  • spread across nodes where possible.

Job timings

• Recording: each job's Context has a Timer (podblock/timing.py). Steps that finish are saved in a new timings table (migration 17) and kept 90 days. Only a job that finishes gets a whole-job row.

• Process jobs:

  • download, fingerprint, cut, waveform;
  • the wait for another job to finish finding repeats;
  • finding repeats, broken into: loading fingerprints, comparing pairs, storing new blocks, rescans, splitting blocks, dropping nested and short blocks, tagging bumpers, reports.

  Re-cuts are saved as kind recut, so they can be told apart from first downloads.

• Edit and unlearn jobs: the find-repeats work and the wait for it.

• Captions jobs: sending the audio, waiting for a free transcriber (with the number of tries), transcribing, and who speaks when. The service now sends {"working": true} when an episode's turn comes, so time spent waiting isn't counted as decoding. Older clients ignore the new line. The service also logs each episode's own times.

• Page: /timings is admin-only and linked from Jobs. For 7, 30 or 90 days, it shows per kind of job and step: runs, median, slowest 10%, time per hour of audio, and total. It also lists the latest jobs step by step. /timings.csv has every row.

Testing

• pytest: 231 passed. ruff check: clean.
• New tests cover: busy replicas answering at once; hanging up and retrying with patience, and waiting without it; a stopping replica sending away waiting requests and refusing new ones; one retry after a cut-off; the speaker-model download; the number of captions threads; timings for process, recut and captions jobs; failed jobs keeping the steps they finished; the timings page and CSV (admin only); pruning old timings.
• The whisper server's real start-up and SIGTERM path, run with a stand-in faster-whisper: the episode in progress finished, the waiting request was sent away, and the process exited 0.
• The timings page was checked by screenshot at 1200 px and 390 px wide, with no sideways scrolling.
• Not tested: the manifests on a real cluster (no kubectl here), and a real faster-whisper model.

---

Generated by Claude Code