Skip to content

Talk mode: real-time voice conversation with barge-in (Ctrl+Alt+T) - #23

Open
juw12 wants to merge 2 commits into
mainfrom
talk-mode
Open

juw12 wants to merge 2 commits into
mainfrom
talk-mode

Conversation

@juw12

@juw12 juw12 commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Talk mode: a real-time spoken conversation with Buddy (Ctrl+Alt+T)

Press Ctrl+Alt+T (or Talk with Buddy in the tray) and talk, no key to hold.

  • Knows when you are done: keyit_talk.TurnTaker reads the words at every pause and waits 0.3 s after a complete
    sentence, 0.7 s when it cannot tell, 1.6 s when you trail off ("and", "because", "um"). English, Indonesian and Chinese
    rules; other languages get the plain 0.7 s pause.
  • Speaks mid-answer: the voice starts on the first sentence while the brain is still writing the rest.
  • Barge-in: EchoGate compares the microphone with what Buddy is playing and stops the voice when you talk over it.
  • Ends with Ctrl+Alt+T, Esc or "goodbye". Push-to-talk and dictation are unchanged.

Gates (frozen rubric buddy-talk-v1)

gate result
G1 tests pass locally PASS: 203 passed (main: 177 passed, 1 skipped)
G2 CI green on this PR PASS: core and windows
G3 barge-in, onset to stop requested, p95 <= 500 ms PASS in simulation (<= 0.2 s) with a headset, or when the user is 3x (natural voice) / 4x (Windows voice) louder than the echo. Stop on the real sound device: not measured
G4 one decision, p95 <= 300 ms FAIL: 0.87 s with Whisper at about 90% CPU load (rules alone: 0.005 ms)
G5 balanced accuracy >= 85% and above fixed silence at the same delay PASS on the audio timeline: 85.4% vs 50.0% (48 synthetic clips, a 0.4 point margin). With compute time under this load: 72.2%, about equal to the best fixed pause (72.9%)
G6 zero false barge-ins at >= 3 echo levels PASS in simulation: 0 in 24 min, 4 levels x 2 voices
G7 push-to-talk and dictation unchanged PASS: existing tests
G8 honest claims README states where Buddy does not beat Jev and what was not measured

Not merged automatically: G4 fails. Giving Whisper only the last 1.2 s still takes 0.43 s (p95) at 100% CPU, so it needs a streaming recognizer or a quiet PC. Council score (3 judges): 1.3.2 4.0, this PR 7.62, blind pair 3-0. Jev was not run; its figures are the vendor's published ones.

Not verified

A real person at a real microphone; a real loudspeaker-to-microphone echo; voice stop on the real output device
(bench part d did not start); the talk mode UI running end to end; latency on a quiet PC; brain latency on CPU-only Ollama.

Bench: bench/talk_turns.py, results in bench/results_talk/, write-up in bench/README.md (Talk mode) and
README.md (Talk mode vs Jev).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants