berd-call: hear realtime calls as Berd's Expert does, or live with --allow-direct-agent-speech

src-tauri/crates/berd-call/PROTOCOL.md

@@ -74,8 +74,8 @@ Detailed backend errors are diagnostics on stderr only; protocol rejection and
 fatal messages are sanitized at the stdout boundary.
 `session.tts` describes the voice that speaks replies, and `set_tts_settings`
 changes it: in Expert-Spokesperson sessions, that is the configured TTS backend
-with `--expert-voice tts` and otherwise the Spokesperson's own voice. With
-`--expert-voice tts`, `session.spokesperson` is the Spokesperson's own OpenAI
+with `--allow-direct-agent-speech` and otherwise the Spokesperson's own voice. With
+`--allow-direct-agent-speech`, `session.spokesperson` is the Spokesperson's own OpenAI
 snapshot, with its own revision, and `set_spokesperson_settings` changes it the
 way `set_tts_settings` changes the Spokesperson's voice in other sessions. A reply in
 the Expert's own voice reports the same `admitted`, `speech_completed`,
@@ -206,7 +206,7 @@ it never admits a replacement while old host audio may still be active.
 {"type":"set_spokesperson_settings","id":u64,"expected_revision":u64,"settings":TtsSettings}
 {"type":"set_input_during_tts","id":u64,"expected_revision":u64,"policy":"allow_barge_in"|"suppress_input"}
 {"type":"reset_input","id":u64}
-{"type":"prepare_speak","id":u64,"acknowledgement":u64|null,"text":string,"resolved_handoff_ids":[string]?,"voice":string?,"language":string?}
+{"type":"prepare_speak","id":u64,"acknowledgement":u64|null,"text":string,"resolved_handoff_ids":[string]?,"voice":string?,"language":string?,"relay":bool?}
 {"type":"output_ready","id":u64,"speech_id":u64}
 audio acknowledgements listed above
 {"type":"query_state","id":u64,"after":u64}
@@ -219,7 +219,16 @@ Unknown fields are rejected. IDs are positive. Speak text is at most 16 KiB.
 `prepare_speak.voice` speaks that one utterance in another voice of the reply
 backend, keeping its other settings; `language` changes a Siri voice's locale
 too. A voice the backend doesn't offer, or a voice for a call whose replies
-are spoken by the Spokesperson, is `not_admitted(invalid_voice)`.
+are spoken by the Spokesperson, is `not_admitted(invalid_voice)`. In a call
+whose replies have their own voice, a reply needs the same `acknowledgement`
+as a directive. With `relay: true` (what
+`berd-call message-spokesperson` sends), the reply is a directive in any
+Expert-Spokesperson call, even one whose replies have their own voice: the
+Spokesperson says it, on the same `acknowledgement` and `resolved_handoff_ids`
+terms. A call whose replies have their own voice holds no handoff open: the
+Expert still hears each one, but `unresolved_handoff_ids` stays empty, no
+reminder or renewal waits on them, and naming one in `resolved_handoff_ids` is
+`not_admitted(invalid_handoff)`.
 The parent cannot author speaking state or finalized input; those are derived
 only from PCM by the child runtime.
 
@@ -254,6 +263,11 @@ linearization point. A speech reservation holds a configuration lease: speech
 admitted before the response retains its old backend/settings, while later
 admission receives the new revision. Pocket's public model identifier cannot be
 changed without selecting and validating another bundle at process startup.
+In a `berd-call start` realtime call, a Spokesperson voice or rate change that
+the conversation overtakes while its new session is being built is not
+rejected: the candidate is discarded and the change is tried again, with the
+conversation as it then stands, once the call is idle. Its one result comes
+when it finally applies or fails.
 
 `set_input_during_tts` changes the policy for later admissions. Its expected
 revision must be positive. The correlated result is always authoritative:

src-tauri/crates/berd-call/README.md

@@ -68,16 +68,14 @@ without changing the call's voice, so agents sharing a call can sound distinct.
 server listens only on `127.0.0.1:5222` by default; all four commands accept
 `--port` when more than one local call must be addressed. `--stream` writes one
 TSV row per delivered transcript event after the `cursor`, `role`, and `text`
-header. In Expert-Spokesperson mode the Expert, like a GPT-Live delegate, hears
-the conversation only when the Spokesperson hands off: the stream prints
-nothing until then, and then every row since the last handoff, ending with the
-`handoff` row. `--stream live` instead shows the conversation as it happens: a
+header. In Expert-Spokesperson mode the stream hears the call as Berd's Expert
+does: a batch of every row since the last one after each Spokesperson reply,
+and at each handoff, ending with the `handoff` row. `--stream live` instead shows the conversation as it happens: a
 `user` row as soon as the words are transcribed, and a `spokesperson_generated`
 row with cursor `0` as soon as a reply is generated, before it is spoken. The
 `spokesperson` or `spokesperson_interrupted` row after playback carries the
 reply's cursor and what was heard. A conventional call has no handoffs, so it
-always streams each row. A `--codex` task, like `--stream`, wakes only at
-handoffs. The standalone host intentionally uses only the default devices. It
+always streams each row. A `--codex` task gets the same batches as `--stream`. The standalone host intentionally uses only the default devices. It
 suppresses recognition during playback when the default output
 identifies itself as built-in speakers, preventing the call from transcribing
 its own speech; headphone-like outputs retain barge-in.
@@ -146,12 +144,12 @@ berd-call session --tts-backend pocket --model-dir /path/to/native-voice-v2 --vo
 berd-call session --stt-backend parakeet --stt-model-dir /path/to/parakeet
 berd-call session --stt-backend openai
 berd-call session --tts-backend openai --mode expert-spokesperson --realtime-url ws://127.0.0.1:18870/v1/realtime
-berd-call session --voice Aaron --language en-US --mode expert-spokesperson --expert-voice tts
+berd-call session --voice Aaron --language en-US --mode expert-spokesperson --allow-direct-agent-speech
 berd-call session --tts-backend openai --tts-url https://proxy.example/v1/audio/speech
 berd-call session --stt-backend openai --stt-url wss://proxy.example/v1/realtime?intent=transcription
 ```
 
-By default the Spokesperson reads the Expert's replies aloud in its own voice. With `--expert-voice tts`, the Expert speaks for itself: berd-call plays each reply through the configured TTS backend, taking turns with the Spokesperson's audio, and your speech interrupts either of them. Afterwards the Spokesperson gets a silent note of what you heard, so it can follow up without repeating it. A reply settles the handoffs it answers once you hear any of it. `--voice`, `--rate`, `status` and `settings --tts` describe and change the Expert's voice. The Spokesperson keeps its own, shown as `session.spokesperson`: choose it with `--spokesperson-voice NAME` and `--spokesperson-rate 0.25–1.5`, at `start` or live with `settings`, and the menu bar offers both voices. Saved settings keep the Spokesperson's voice for the next call whose replies have their own, but not `--expert-voice`, so pass that on each `start`.
+In an Expert–Spokesperson call, the Expert reaches you through the Spokesperson: `berd-call message-spokesperson --re CURSOR [--resolves HANDOFF_ID]... TEXT` has it tell you in its own voice. Like the directives of Berd's Expert, the message must acknowledge the latest input (the Spokesperson's own speech doesn't count), or it comes back `spoke:false` with what was missed, and a handoff it names settles once the Spokesperson has said it. `speak` is refused unless the call was started with `--allow-direct-agent-speech`. With it, the Expert can also speak for itself: berd-call plays each `speak` reply through the configured TTS backend, taking turns with the Spokesperson's audio, and your speech interrupts either of them. Afterwards the Spokesperson gets a silent note of what you heard, so it can follow up without repeating it. A reply has to acknowledge everything you've said, as `speak --re` always does. Such a call holds no handoff open: each still reaches the Expert as a `handoff` row, but nothing waits for an answer, reminds the Expert, or holds back renewing or recovering the Spokesperson's session, and there is nothing for `--resolves` to name. `--voice`, `--rate`, `status` and `settings --tts` describe and change the Expert's voice. The Spokesperson keeps its own, shown as `session.spokesperson`: choose it with `--spokesperson-voice NAME` and `--spokesperson-rate 0.25–1.5`, at `start` or live with `settings`, and the menu bar offers both voices. Saved settings keep the Spokesperson's voice for the next call whose replies have their own, but not `--allow-direct-agent-speech`, so pass that on each `start`.
 
 The three URL flags take full endpoints and override only their matching service for that call. Expert–Spokesperson uses `--realtime-url` for both input and output; `--stt-url` applies only to conventional mode, and `--tts-url` also applies to the Expert's own voice. Without an override, each service uses its OpenAI endpoint (or its existing environment override). `OPENAI_API_KEY` supplies the CLI credential; the desktop app stores keys separately by endpoint URL in Keychain.
 

src-tauri/crates/berd-call/src/cli_help.rs

@@ -8,6 +8,7 @@ Usage:
 Call runtime:
   start                   Run a microphone-backed local voice call
   speak                   Speak through the active local call
+  message-spokesperson    Have the Spokesperson tell the user
   status                  Inspect the active local call
   catch-up                Read finalized input after a cursor
   wait-for-input          Wait for finalized input after a cursor
@@ -33,17 +34,17 @@ Run `berd-call help <command>` for command-specific usage."#;
 const START_HELP: &str = r#"Run a local voice call backed by the shared session runtime.
 
 Usage:
-  berd-call start [--port PORT] [--stream [handoffs|live] | --codex] [--non-blocking]
+  berd-call start [--port PORT] [--stream [live] | --codex] [--non-blocking]
     [--no-menu-bar]
     --voice NAME --language BCP47 [session options]
 
 The foreground process owns the default microphone and audio output. Session
 options are the same as `berd-call session`, except the PCM descriptor is owned
 internally. `--stream` prints cursor, role, and text as TSV. In
-Expert-Spokesperson mode it prints nothing until the Spokesperson hands off,
-then everything since the last handoff; `--stream live` prints each row as it
-happens. `--codex` delivers the handoff batches to the calling Codex task
-(requires CODEX_THREAD_ID). A
+Expert-Spokesperson mode it prints a batch after every Spokesperson reply and
+at each handoff, as Berd's Expert hears the call; `--stream live` prints each
+row as it happens. `--codex` delivers the same batches as `--stream` to the
+calling Codex task (requires CODEX_THREAD_ID). A
 menu-bar item controls speech rate, input muting, and ending the call; pass
 `--no-menu-bar` to omit it.
 
@@ -65,7 +66,23 @@ for delivery. Non-blocking sessions require --stream or --codex and emit speech_
 rows only for interruptions, failures, or pending user input. Interrupted results include
 estimatedSpokenText, a best-effort prefix based on delivered audio. --voice speaks
 this text in another voice of the call's reply backend without changing its saved
-voice, so several agents can share one call in distinct voices."#;
+voice, so several agents can share one call in distinct voices. An
+Expert-Spokesperson call refuses speak unless started with
+--allow-direct-agent-speech; otherwise use message-spokesperson. With it, a
+reply acknowledging the latest input is spoken in the Expert's own voice, and
+handoffs are never held open, so there is nothing for --resolves to name."#;
+
+const MESSAGE_SPOKESPERSON_HELP: &str = r#"Message the Spokesperson of the active realtime call.
+
+Usage:
+  berd-call message-spokesperson [--port PORT] --re CURSOR
+    [--resolves HANDOFF_ID]... TEXT
+
+The Spokesperson tells the user in its own voice, even in a call with
+--allow-direct-agent-speech, where `speak` uses the Expert's. Like a directive
+from Berd's Expert, the message must acknowledge the latest input (the
+Spokesperson's own speech doesn't count), each --resolves settles that handoff
+once the Spokesperson has said it, and the result is the same as `speak`'s."#;
 
 const SETTINGS_HELP: &str = r#"Update settings for the active local voice call.
 
@@ -81,7 +98,7 @@ Usage:
   berd-call settings [--port PORT] --restart [session options]
 
 The setting applies to subsequent speak calls. In a call started with
---expert-voice tts, --voice and --rate change the agents' voice, and
+--allow-direct-agent-speech, --voice and --rate change the agents' voice, and
 --spokesperson-voice and --spokesperson-rate change the Spokesperson's (an
 OpenAI realtime voice, 0.25 to 1.5). Non-blocking delivery requires
 a call started with --stream or --codex. Successful delivery produces no notification."#;
@@ -120,7 +137,7 @@ Usage:
 Each form also accepts:
     [--rate FLOAT]
     [--stt-backend macos|parakeet|openai] [--stt-model-dir PATH]
-    [--mode conventional|expert-spokesperson] [--expert-voice tts|spokesperson]
+    [--mode conventional|expert-spokesperson] [--allow-direct-agent-speech]
     [--spokesperson-voice NAME] [--spokesperson-rate 0.25..1.5]
     [--realtime-url WS_URL] [--stt-url WS_URL] [--tts-url HTTP_URL]
 
@@ -129,9 +146,9 @@ agent integration. See PROTOCOL.md for the framed stdin, stdout, and PCM
 contracts. Endpoint URL flags override only the matching service for this call
 and default to OpenAI when no environment override is set.
 
-In expert-spokesperson mode, --expert-voice tts speaks the Expert's replies
-through the configured TTS backend, in its own voice beside the Spokesperson's,
-instead of having the Spokesperson read them aloud. --spokesperson-voice and
+In expert-spokesperson mode, the Expert speaks through the Spokesperson.
+--allow-direct-agent-speech also lets it speak to the user itself, through the
+configured TTS backend, in its own voice beside the Spokesperson's. --spokesperson-voice and
 --spokesperson-rate then choose the Spokesperson's own voice and speed."#;
 
 const SYNTHESIZE_HELP: &str = r#"Render text through a configured TTS backend into a new WAV file.
@@ -211,6 +228,7 @@ fn help_for(topic: &[&str]) -> Option<&'static str> {
         [] => Some(TOP_LEVEL_HELP),
         ["start"] => Some(START_HELP),
         ["speak"] => Some(SPEAK_HELP),
+        ["message-spokesperson"] => Some(MESSAGE_SPOKESPERSON_HELP),
         ["status"] => Some(STATUS_HELP),
         ["catch-up"] | ["wait-for-input"] => Some(POLL_INPUT_HELP),
         ["settings"] => Some(SETTINGS_HELP),

src-tauri/crates/berd-call/src/host_control.rs

@@ -30,6 +30,7 @@ pub(crate) trait HostControl: Send + Sync + 'static {
         resolved_handoff_ids: Vec<String>,
         voice: Option<String>,
         language: Option<String>,
+        relay: bool,
     ) -> Result<Value, String>;
     fn stop(&self) -> Result<Value, String>;
     fn set_non_blocking(&self, enabled: bool) -> Result<Value, String>;
@@ -121,6 +122,8 @@ pub(crate) enum ControlRequest {
         voice: Option<String>,
         #[serde(default, skip_serializing_if = "Option::is_none")]
         language: Option<String>,
+        #[serde(default, skip_serializing_if = "std::ops::Not::not")]
+        relay: bool,
     },
     Stop,
     NonBlocking {
@@ -239,14 +242,25 @@ fn handle_connection(mut stream: TcpStream, control: &dyn HostControl) -> Result
                     resolved_handoff_ids,
                     voice,
                     language,
+                    relay,
                 } => {
                     validate_speak(
                         &text,
                         &resolved_handoff_ids,
                         voice.as_deref(),
                         language.is_some(),
                     )?;
-                    control.speak(text, acknowledgement, resolved_handoff_ids, voice, language)
+                    if relay && voice.is_some() {
+                        return Err("a message to the Spokesperson takes no voice".into());
+                    }
+                    control.speak(
+                        text,
+                        acknowledgement,
+                        resolved_handoff_ids,
+                        voice,
+                        language,
+                        relay,
+                    )
                 }
                 ControlRequest::Stop => control.stop(),
                 ControlRequest::TtsSettings { settings } => control.set_tts(settings),
@@ -413,13 +427,15 @@ mod tests {
             resolved_handoff_ids: Vec<String>,
             voice: Option<String>,
             language: Option<String>,
+            relay: bool,
         ) -> Result<Value, String> {
             Ok(json!({
                 "spoken":text,
                 "acknowledgement":acknowledgement,
                 "resolvedHandoffIds":resolved_handoff_ids,
                 "voice":voice,
                 "language":language,
+                "relay":relay,
             }))
         }
 
@@ -468,6 +484,7 @@ mod tests {
                     resolved_handoff_ids: vec!["call-1".into()],
                     voice: Some("Nora".into()),
                     language: Some("en-GB".into()),
+                    relay: false,
                 }
             )
             .unwrap(),
@@ -477,6 +494,7 @@ mod tests {
                 "resolvedHandoffIds":["call-1"],
                 "voice":"Nora",
                 "language":"en-GB",
+                "relay":false,
             })
         );
         assert_eq!(