Repository navigation
berd-call: hear realtime calls as Berd's Expert does, or live with --allow-direct-agent-speech - #393
Closed
johnmatthewtennant wants to merge 24 commits into
Closed
johnmatthewtennant wants to merge 24 commits into
johnmatthewtennant wants to merge 24 commits into
Conversation
johnmatthewtennant
changed the title
johnmatthewtennant
changed the title
johnmatthewtennant
changed the title
johnmatthewtennant
force-pushed
the
jtennant/berd-call-simple-handoffs
branch
2 times, most recently
from
October 20, 2026 15:41
6e2d467 to
731bc7d
Compare
johnmatthewtennant
marked this pull request as ready for review
October 20, 2026 16:14
With --expert-voice tts, a reply could be dropped: one written before the user's latest words came back pending, and one naming a handoff already settled was not admitted. An open handoff also kept the Spokesperson's session from renewing, and ended the call if the provider expired it. Such a reply is now always admitted, whatever its cursor, and settles every open handoff as soon as it is, so --resolves is ignored. Open handoffs no longer hold back renewal, recovery or a voice change, and the replacement session is told what they asked so it doesn't hand them off again. A reply that arrives during a replacement waits for it rather than failing it. Calls whose replies the Spokesperson says, and the Berd app, behave as before. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
In a call with --expert-voice tts, every speak played in the Expert's voice, so an Expert couldn't have the Spokesperson tell the user something instead. `berd-call message-spokesperson TEXT` (protocol: `prepare_speak.relay`) hands the text to the Spokesperson to say, on the same terms as an own-voice reply there: admitted however stale, and answering every open handoff once admitted. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
`message-spokesperson` is how an Expert answers through the Spokesperson, so it now works the same in a call without --expert-voice tts: admitted however much the user has said since, and answering every open handoff, with no --re or --resolves. `speak` there keeps today's rules. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
…nt-speech In an Expert-Spokesperson call, the Expert now reaches the user through the Spokesperson with `message-spokesperson`, and `speak` is refused unless the call was started with --allow-direct-agent-speech. That flag replaces --expert-voice tts. With it, a reply in the Expert's own voice must acknowledge the latest input, as `speak --re` does everywhere, and once admitted it still answers every open handoff. A message to the Spokesperson stays admitted however stale. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
A `--codex` task heard a realtime call only at handoffs. It now gets a batch after every Spokesperson reply and at each handoff, as Berd's Expert and berd-call before the handoffs stream did, so it can follow the conversation and answer what was said last. `--stream` (handoffs) still holds rows until a handoff, and `--stream live` is unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
`message-spokesperson` was admitted however stale and answered every open handoff. It now takes the terms of a directive from Berd's Expert: `--re CURSOR` is required and must acknowledge the latest input (the Spokesperson's own speech doesn't count), or the message comes back `spoke:false` with what was missed, and each `--resolves HANDOFF_ID` settles that handoff once the Spokesperson has said it. This holds in any Expert-Spokesperson call, including one with --allow-direct-agent-speech, where `speak` still answers every handoff. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
`--stream` (or `--stream handoffs`) printed nothing in an Expert-Spokesperson call until a handoff. It now prints the same batches as `--codex`, after every Spokesperson reply and at each handoff, so the CLI has two ways to hear a realtime call: as Berd's Expert does, or as it happens with `--stream live`. `handoffs` is no longer accepted after `--stream`. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
With --allow-direct-agent-speech the Spokesperson still hands off and the Expert still hears each handoff, but nothing tracks it: no reminders, no renewal or recovery held back, no note for a replacement session, and nothing for --resolves to name. A reply is said once it acknowledges the latest input, as before. Calls without the flag keep Berd's handoff rules. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
The voice session read its arguments in pairs to find --pcm-output-fd, which the host appends last. The valueless --allow-direct-agent-speech shifted the pairs, so every call started with it failed with "voice session event stream closed during startup". Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
A Spokesperson voice or rate change builds a new realtime session, and if anyone spoke before it was ready, the change was rejected with "Spokesperson conversation changed during voice replacement", so a menu-bar change made mid-conversation never took. In a berd-call start call it is now tried again once the call is idle, with the conversation as it then stands, and reports one result when it applies. The Berd app's handling is unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
OpenAI sends the Spokesperson's audio about as fast as it plays. Playback sent it to the host only in whole 170 ms chunks and started on the first one, so the speaker often ran dry before the next chunk was complete, and any late delta left an audible gap. Gather 300 ms of each reply before it plays, then send each delta as it arrives. A test plays a reply arriving in real time with one 150 ms stall: whole chunks leave a 109 ms silence; the lead plays through. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
On a live realtime call, in both modes, change the Spokesperson's speed, then its voice, then the input policy, then mute and unmute, and check each one lands. With a separate agent voice, also change the agent's voice and speed. Check that the Spokesperson's realtime session is rebuilt with each new voice and speed. Run the retry test for a change the user overtook in both modes as well. With a separate agent voice, the Spokesperson rate change it retries fails without the retry. On the host, a Spokesperson voice or rate change keeps the other setting and the revision, and is refused when the call has no separate Spokesperson voice. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
…dio" This reverts commit aad01ee. Its premise was wrong: measured on six live replies, OpenAI Realtime sends a reply's audio 3 to 5.5 times faster than it plays, in 150 to 400 ms deltas, so a lead covers no late arrival and only delays the first sound. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
The host reports a chunk played only when the output device has played it: about 60 ms after it is heard on a MacBook's speakers, and a few hundred more on Bluetooth headphones. Playback held at most 400 ms beyond what was reported played (less 100 ms of delivery safety), so on headphones the player ran dry while waiting for credit, and the Spokesperson's speech, and an agent's, came out in short gaps. Let the host hold a second beyond its reports, which plays through reports up to about 800 ms late. A test plays 3 s of audio with reports 300 ms late: the old limit leaves 1041 ms of silence; this leaves none. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
When the host's player stops, as when the output device changes, the host ends that speech and reports audio_failed. The session can still send chunk, end or cancel records for it before hearing the report. The host treated those as protocol errors and ended the call. It now drops late chunk and end records for the speech it retired and answers a late cancel with what it played. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
A Spokesperson reply whose host player stopped ended the whole call with "host audio output failed". The session now ends that reply where the host stopped, as if the user had interrupted it: the provider's item is truncated to what played, the transcript is published as interrupted, and the next reply gets a new player. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: John Tennant <[email protected]>
Switching the output device restarts the microphone and resets the Spokesperson's input. The realtime loop waited for the provider to confirm the clear, so microphone audio stopped draining, the 32-frame session queue filled, and the session failed with "session PCM input queue is full". Input resets and mutes now send the clear without waiting. The loop answers them in order once the provider confirms, coalesces a request that arrives while a clear is outstanding into one follow-up clear, and holds a pending voice change until no clear is outstanding. Resets share the input queue with microphone audio, so a clear discards exactly the audio sent before it. Signed-off-by: John Tennant <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
The tests restacked from below still started sessions with `--expert-voice tts` and built speech requests without `relay`. Signed-off-by: John Tennant <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
In Berd mode, a message-spokesperson that arrived while a Spokesperson voice change was being built was held for the change, but the held message made the change unsafe. The change was rejected, queued and restarted at once, so neither could ever go through. Let a held directive go before a queued change starts again, and keep queued changes from holding directives back. Signed-off-by: John Tennant <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
A reset that arrived while another clear was outstanding was folded into one follow-up clear sent only after the first was confirmed, so audio heard between the two controls survived the second reset. Send every clear where its control arrives and answer the queued controls once the provider has confirmed them all, in order. Signed-off-by: John Tennant <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: John Tennant <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
When the host's player stops, the Spokesperson playback worker sends its result and stops taking audio. Provider audio that arrived in the meantime hit the closed worker first and ended the call, before the loop read the result that interrupts the reply where it was heard. Drop that audio instead and let the result settle the reply. Signed-off-by: John Tennant <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
The output_ready gate and failure tests restacked from below used the reply's old optional text and started sessions with `--expert-voice tts`. A call that lets the agent speak holds no handoff open, so the failure tests now check only what the Spokesperson is told. Signed-off-by: John Tennant <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
johnmatthewtennant
force-pushed
the
jtennant/berd-call-simple-handoffs
branch
from
October 20, 2026 17:21
7183543 to
0fc4238
Compare
Contributor
Author
|
🤖 Combined into #397 with its commits kept, so the stack no longer needs restacking and review covers one diff. Open review findings carry over there. (AI-generated comment) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An agent in a realtime (Expert–Spokesperson) Berd Call couldn't take part the way Berd's own Expert does:
--codexand--streamheld the call back until a handoff, so the agent missed what the Spokesperson said in between and couldn't add to or correct it.speakplayed in the agent's voice, so it had no way to answer through the Spokesperson and keep the call one voice.The Buzz plugin needs the opposite: it sees the call live, and agents speak for themselves. There, Berd's handoff tracking only got in the way. Handoffs blocked renewal, a provider expiry with one open ended the call, and a reply naming a settled handoff was rejected.
Realtime calls now run in one of two modes.
Berd mode (the default, for
--codexand--stream) hears the call as Berd's Expert does:--codexand--streamget the same batches, and--stream handoffsis gone.berd-call message-spokesperson --re CURSOR [--resolves HANDOFF_ID]... TEXT(protocol:prepare_speak.relay) is a directive. The Spokesperson says it, and it settles the handoffs it names once heard. Like any directive, it must acknowledge the latest input (the Spokesperson's own speech doesn't count), or it comes backspoke:falsewith what was missed.speakis refused, pointing tomessage-spokesperson.Buzz mode (
--stream live --allow-direct-agent-speech, which replaces--expert-voice tts) is for the plugin:speak --re CURSORplays in the agent's own voice, and must acknowledge the latest input. A reply that arrives while the Spokesperson's session is being replaced waits for it.message-spokespersonworks as in Berd mode.--resolvesisnot_admitted(invalid_handoff).In a
berd-call startrealtime call, a Spokesperson voice or rate change that someone talks over while its new session is being built is no longer rejected with "Spokesperson conversation changed during voice replacement". It is tried again once the call is idle, with the conversation as it then stands.Clearing the Spokesperson's input buffer no longer stalls the call while the provider confirms the clear. Microphone audio keeps flowing during the clear, and it stays in order with it: input sent before a clear is discarded, and input sent after is kept. This used to crash realtime calls when the audio devices changed.
The Berd app doesn't use these paths, so its behavior is unchanged.
Related issue
None found.
Testing
speakis refused before reaching the session, andmessage-spokespersongoes through.message-spokespersoncomes backpending. Acknowledging the latest and naming the first, it is admitted and reaches the Spokesperson as something to say, and only the second stays open.pending. Acknowledging the latest, it is admitted and spoken, and no handoff is open.microphone_audio_keeps_flowing_while_the_provider_confirms_an_input_reset).