Skip to content

Repository navigation

berd-call: interrupt speech whose player stops instead of failing it - #394

Closed
johnmatthewtennant wants to merge 25 commits into
jtennant/berd-call-simple-handoffsfrom
jtennant/berd-call-output-device-change
Closed

berd-call: interrupt speech whose player stops instead of failing it#394
johnmatthewtennant wants to merge 25 commits into
jtennant/berd-call-simple-handoffsfrom
jtennant/berd-call-output-device-change

Conversation

@johnmatthewtennant

@johnmatthewtennant johnmatthewtennant commented Oct 20, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Switching speakers while a call is speaking stops macOS's audio engine, so the host reports that its player failed. In a conventional call this ended the speech as speech_failed with host audio output failed, and berd-call speak printed that as an error even though the call kept going.

Now a host failure that leaves the output quiet ends the speech as speech_interrupted at the point it was heard, the same as talking over it. The next reply plays on the new device. Expert-Spokesperson calls already handled this case the same way.

Related issue

N/A. No existing issue was found. A user hit this in a live call by switching from AirPods to the MacBook Pro speakers.

Testing

A conventional session with OpenAI speech has its host player stop partway through the first reply (speech_whose_player_stops_is_interrupted_where_it_was_heard_and_the_call_goes_on).

  • Before the fix: {"type":"speech_failed","message":"host audio output failed"}
  • After the fix: speech_interrupted before the end of the text, and the next prepare_speak completes.

@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/berd-call-simple-handoffs branch from b3cc5cc to 6e2d467 Compare October 20, 2026 15:29
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/berd-call-output-device-change branch from e751b15 to 49d39c0 Compare October 20, 2026 15:29
johnmatthewtennant and others added 21 commits October 20, 2026 11:39
@johnmatthewtennant @claude
berd-call: let any reply in the Expert's own voice answer every handoff
b538358 
With --expert-voice tts, a reply could be dropped: one written before
the user's latest words came back pending, and one naming a handoff
already settled was not admitted. An open handoff also kept the
Spokesperson's session from renewing, and ended the call if the
provider expired it.

Such a reply is now always admitted, whatever its cursor, and settles
every open handoff as soon as it is, so --resolves is ignored. Open
handoffs no longer hold back renewal, recovery or a voice change, and
the replacement session is told what they asked so it doesn't hand
them off again. A reply that arrives during a replacement waits for
it rather than failing it. Calls whose replies the Spokesperson says,
and the Berd app, behave as before.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: let the Expert message the Spokesperson in an own-voice call
e320f97 
In a call with --expert-voice tts, every speak played in the Expert's
voice, so an Expert couldn't have the Spokesperson tell the user
something instead. `berd-call message-spokesperson TEXT` (protocol:
`prepare_speak.relay`) hands the text to the Spokesperson to say, on the
same terms as an own-voice reply there: admitted however stale, and
answering every open handoff once admitted.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: let a message to the Spokesperson answer handoffs in any call
b359ffa 
`message-spokesperson` is how an Expert answers through the Spokesperson,
so it now works the same in a call without --expert-voice tts: admitted
however much the user has said since, and answering every open handoff,
with no --re or --resolves. `speak` there keeps today's rules.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: let the Expert speak directly only with --allow-direct-age…
9d52b57 
…nt-speech

In an Expert-Spokesperson call, the Expert now reaches the user through
the Spokesperson with `message-spokesperson`, and `speak` is refused
unless the call was started with --allow-direct-agent-speech. That flag
replaces --expert-voice tts. With it, a reply in the Expert's own voice
must acknowledge the latest input, as `speak --re` does everywhere, and
once admitted it still answers every open handoff. A message to the
Spokesperson stays admitted however stale.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: let a Codex task hear the call as Berd's Expert does
02419fc 
A `--codex` task heard a realtime call only at handoffs. It now gets a
batch after every Spokesperson reply and at each handoff, as Berd's
Expert and berd-call before the handoffs stream did, so it can follow
the conversation and answer what was said last. `--stream` (handoffs)
still holds rows until a handoff, and `--stream live` is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: make a message to the Spokesperson an Expert directive
9056bf1 
`message-spokesperson` was admitted however stale and answered every
open handoff. It now takes the terms of a directive from Berd's Expert:
`--re CURSOR` is required and must acknowledge the latest input (the
Spokesperson's own speech doesn't count), or the message comes back
`spoke:false` with what was missed, and each `--resolves HANDOFF_ID`
settles that handoff once the Spokesperson has said it. This holds in
any Expert-Spokesperson call, including one with
--allow-direct-agent-speech, where `speak` still answers every handoff.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: stream realtime calls the way Berd's Expert hears them
bba40eb 
`--stream` (or `--stream handoffs`) printed nothing in an
Expert-Spokesperson call until a handoff. It now prints the same
batches as `--codex`, after every Spokesperson reply and at each
handoff, so the CLI has two ways to hear a realtime call: as Berd's
Expert does, or as it happens with `--stream live`. `handoffs` is no
longer accepted after `--stream`.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: hold no handoff open when the Expert may speak for itself
b07c208 
With --allow-direct-agent-speech the Spokesperson still hands off and
the Expert still hears each handoff, but nothing tracks it: no
reminders, no renewal or recovery held back, no note for a replacement
session, and nothing for --resolves to name. A reply is said once it
acknowledges the latest input, as before. Calls without the flag keep
Berd's handoff rules.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: find the PCM descriptor after --allow-direct-agent-speech
727a89f 
The voice session read its arguments in pairs to find
--pcm-output-fd, which the host appends last. The valueless
--allow-direct-agent-speech shifted the pairs, so every call started
with it failed with "voice session event stream closed during
startup".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: retry a Spokesperson voice change the conversation overtook
71529d9 
A Spokesperson voice or rate change builds a new realtime session, and
if anyone spoke before it was ready, the change was rejected with
"Spokesperson conversation changed during voice replacement", so a
menu-bar change made mid-conversation never took. In a berd-call start
call it is now tried again once the call is idle, with the
conversation as it then stands, and reports one result when it applies.
The Berd app's handling is unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: gather a lead before playing the Spokesperson's audio
a18f699 
OpenAI sends the Spokesperson's audio about as fast as it plays.
Playback sent it to the host only in whole 170 ms chunks and started
on the first one, so the speaker often ran dry before the next chunk
was complete, and any late delta left an audible gap.

Gather 300 ms of each reply before it plays, then send each delta as it
arrives. A test plays a reply arriving in real time with one 150 ms
stall: whole chunks leave a 109 ms silence; the lead plays through.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: test changing every setting in a live call
5aa3160 
On a live realtime call, in both modes, change the Spokesperson's
speed, then its voice, then the input policy, then mute and unmute,
and check each one lands. With a separate agent voice, also change the
agent's voice and speed. Check that the Spokesperson's realtime session
is rebuilt with each new voice and speed.

Run the retry test for a change the user overtook in both modes as
well. With a separate agent voice, the Spokesperson rate change it
retries fails without the retry. On the host, a Spokesperson voice or
rate change keeps the other setting and the revision, and is refused
when the call has no separate Spokesperson voice.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
Revert "berd-call: gather a lead before playing the Spokesperson's au…
37646e7 
…dio"

This reverts commit aad01ee. Its premise was wrong: measured on six
live replies, OpenAI Realtime sends a reply's audio 3 to 5.5 times
faster than it plays, in 150 to 400 ms deltas, so a lead covers no
late arrival and only delays the first sound.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: keep a second of audio ahead of the host's played reports
cfc97b1 
The host reports a chunk played only when the output device has played
it: about 60 ms after it is heard on a MacBook's speakers, and a few
hundred more on Bluetooth headphones. Playback held at most 400 ms
beyond what was reported played (less 100 ms of delivery safety), so
on headphones the player ran dry while waiting for credit, and the
Spokesperson's speech, and an agent's, came out in short gaps.

Let the host hold a second beyond its reports, which plays through
reports up to about 800 ms late. A test plays 3 s of audio with
reports 300 ms late: the old limit leaves 1041 ms of silence; this
leaves none.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: let the host ignore records for speech it already ended
c4090c2 
When the host's player stops, as when the output device changes, the
host ends that speech and reports audio_failed. The session can still
send chunk, end or cancel records for it before hearing the report.
The host treated those as protocol errors and ended the call. It now
drops late chunk and end records for the speech it retired and answers
a late cancel with what it played.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
berd-call: keep a realtime call going when the host's player stops
acf9487 
A Spokesperson reply whose host player stopped ended the whole call
with "host audio output failed". The session now ends that reply where
the host stopped, as if the user had interrupted it: the provider's
item is truncated to what played, the transcript is published as
interrupted, and the next reply gets a new player.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]>
@johnmatthewtennant @claude
Keep microphone input flowing while an input reset clears
23d1202 
Switching the output device restarts the microphone and resets the
Spokesperson's input. The realtime loop waited for the provider to
confirm the clear, so microphone audio stopped draining, the 32-frame
session queue filled, and the session failed with "session PCM input
queue is full".

Input resets and mutes now send the clear without waiting. The loop
answers them in order once the provider confirms, coalesces a request
that arrives while a clear is outstanding into one follow-up clear,
and holds a pending voice change until no clear is outstanding.
Resets share the input queue with microphone audio, so a clear
discards exactly the audio sent before it.

Signed-off-by: John Tennant <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@johnmatthewtennant @claude
Adapt the Expert reply tests to direct agent speech
90945f9 
The tests restacked from below still started sessions with
`--expert-voice tts` and built speech requests without `relay`.

Signed-off-by: John Tennant <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@johnmatthewtennant @claude
Say a held message before retrying the voice change it waited for
39aa568 
In Berd mode, a message-spokesperson that arrived while a Spokesperson
voice change was being built was held for the change, but the held
message made the change unsafe. The change was rejected, queued and
restarted at once, so neither could ever go through. Let a held
directive go before a queued change starts again, and keep queued
changes from holding directives back.

Signed-off-by: John Tennant <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@johnmatthewtennant @claude
Give each input reset its own clear
c48a6dc 
A reset that arrived while another clear was outstanding was folded
into one follow-up clear sent only after the first was confirmed, so
audio heard between the two controls survived the second reset. Send
every clear where its control arrives and answer the queued controls
once the provider has confirmed them all, in order.

Signed-off-by: John Tennant <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@johnmatthewtennant @claude
Adapt the Expert reply interruption tests to direct agent speech
731bc7d 
Signed-off-by: John Tennant <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/berd-call-simple-handoffs branch from 6e2d467 to 731bc7d Compare October 20, 2026 15:41
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/berd-call-output-device-change branch from 49d39c0 to d104117 Compare October 20, 2026 15:43
johnmatthewtennant and others added 4 commits October 20, 2026 11:46
@johnmatthewtennant @claude
Format the input reset test
8c8e8df 
Signed-off-by: John Tennant <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@johnmatthewtennant @claude
Let a stopped playback worker's result settle its reply
7183543 
When the host's player stops, the Spokesperson playback worker sends
its result and stops taking audio. Provider audio that arrived in the
meantime hit the closed worker first and ended the call, before the
loop read the result that interrupts the reply where it was heard.
Drop that audio instead and let the result settle the reply.

Signed-off-by: John Tennant <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@johnmatthewtennant @claude
Interrupt speech whose player stops instead of failing it
0fa7738 
When the speakers change mid-reply, macOS stops the audio engine and
the host reports its player failed. A conventional call reported that
as `speech_failed: host audio output failed`, so `speak` printed an
error even though the call carried on. Treat a quiescent host failure
like a barge-in: the speech ends as interrupted where it was heard.

Signed-off-by: John Tennant <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@johnmatthewtennant @claude
Drop a no-op drop from the player-stop test
e148a3f 
Signed-off-by: John Tennant <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/berd-call-output-device-change branch from d104117 to e148a3f Compare October 20, 2026 15:53
@johnmatthewtennant
johnmatthewtennant marked this pull request as ready for review October 20, 2026 16:14
@johnmatthewtennant
johnmatthewtennant force-pushed the jtennant/berd-call-simple-handoffs branch from 7183543 to 0fc4238 Compare October 20, 2026 17:21
@johnmatthewtennant johnmatthewtennant mentioned this pull request Oct 20, 2026
Merged
@johnmatthewtennant

johnmatthewtennant commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

🤖 Combined into #397 with its commits kept, so the stack no longer needs restacking and review covers one diff. Open review findings carry over there. (AI-generated comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@johnmatthewtennant