berd-call: let the Expert speak in its own voice in realtime calls

src-tauri/crates/berd-call/PROTOCOL.md

@@ -72,6 +72,14 @@ OpenAI snapshots contain `model`, `voice`, and `rate`; Siri contains `voice`,
 and `rate`. Credentials, endpoints, and bundle paths never appear on stdout.
 Detailed backend errors are diagnostics on stderr only; protocol rejection and
 fatal messages are sanitized at the stdout boundary.
+`session.tts` describes the voice that speaks replies, and `set_tts_settings`
+changes it: in Expert-Spokesperson sessions, that is the configured TTS backend
+with `--expert-voice tts` and otherwise the Spokesperson's own voice. With
+`--expert-voice tts`, `session.spokesperson` is the Spokesperson's own OpenAI
+snapshot, with its own revision, and `set_spokesperson_settings` changes it the
+way `set_tts_settings` changes the Spokesperson's voice in other sessions. A reply in
+the Expert's own voice reports the same `admitted`, `speech_completed`,
+`speech_interrupted`, and `speech_failed` lifecycle as a conventional reply.
 `session.input_during_tts` is the authoritative effective assistant-input
 policy and has its own revision. Status-sound settings are per-update parameters,
 not session snapshot configuration. The runtime owns cue cadence and playback; it
@@ -195,9 +203,10 @@ it never admits a replacement while old host audio may still be active.
 {"type":"set_input_muted","id":u64,"active":bool}
 {"type":"set_conversation_status","id":u64,"status":"working"|"waiting","settings":StatusSoundSettings}
 {"type":"set_tts_settings","id":u64,"expected_revision":u64,"settings":TtsSettings}
+{"type":"set_spokesperson_settings","id":u64,"expected_revision":u64,"settings":TtsSettings}
 {"type":"set_input_during_tts","id":u64,"expected_revision":u64,"policy":"allow_barge_in"|"suppress_input"}
 {"type":"reset_input","id":u64}
-{"type":"prepare_speak","id":u64,"acknowledgement":u64|null,"text":string}
+{"type":"prepare_speak","id":u64,"acknowledgement":u64|null,"text":string,"resolved_handoff_ids":[string]?,"voice":string?,"language":string?}
 {"type":"output_ready","id":u64,"speech_id":u64}
 audio acknowledgements listed above
 {"type":"query_state","id":u64,"after":u64}
@@ -207,6 +216,10 @@ audio acknowledgements listed above
 ```
 
 Unknown fields are rejected. IDs are positive. Speak text is at most 16 KiB.
+`prepare_speak.voice` speaks that one utterance in another voice of the reply
+backend, keeping its other settings; `language` changes a Siri voice's locale
+too. A voice the backend doesn't offer, or a voice for a call whose replies
+are spoken by the Spokesperson, is `not_admitted(invalid_voice)`.
 The parent cannot author speaking state or finalized input; those are derived
 only from PCM by the child runtime.
 
@@ -339,7 +352,7 @@ cancellation and shutdown terminals.
 
 ```text
 {"type":"pending","id":u64,"utterances":[{"token":u64,"text":string}]}
-{"type":"not_admitted","id":u64,"reason":"paused"|"in_progress"|"cancelled"|"empty_text"}
+{"type":"not_admitted","id":u64,"reason":"paused"|"in_progress"|"cancelled"|"empty_text"|"invalid_handoff"|"invalid_voice"}
 {"type":"state","id":u64,"confirmed_token":u64,"utterances_after":[{"token":u64,"text":string}]}
 {"type":"cancel_result","id":u64,"outcome":"cancelled"|"stale","speech_id":u64|null}
 {"type":"speech_started","id":u64,"speech_id":u64}

src-tauri/crates/berd-call/README.md

@@ -57,9 +57,13 @@ Start a call with the same backend flags accepted by `session`:
 berd-call start --voice Aaron --language en-US --stream
 berd-call status
 berd-call speak --re 7 "I found the cause."
+berd-call speak --voice Nora "Sol here: the tests pass."
 berd-call stop
 ```
 
+`speak --voice` speaks one reply in another voice of the call's reply backend
+without changing the call's voice, so agents sharing a call can sound distinct.
+
 `start` stays in the foreground and owns the child session process. The control
 server listens only on `127.0.0.1:5222` by default; all four commands accept
 `--port` when more than one local call must be addressed. `--stream` writes one
@@ -142,11 +146,14 @@ berd-call session --tts-backend pocket --model-dir /path/to/native-voice-v2 --vo
 berd-call session --stt-backend parakeet --stt-model-dir /path/to/parakeet
 berd-call session --stt-backend openai
 berd-call session --tts-backend openai --mode expert-spokesperson --realtime-url ws://127.0.0.1:18870/v1/realtime
+berd-call session --voice Aaron --language en-US --mode expert-spokesperson --expert-voice tts
 berd-call session --tts-backend openai --tts-url https://proxy.example/v1/audio/speech
 berd-call session --stt-backend openai --stt-url wss://proxy.example/v1/realtime?intent=transcription
 ```
 
-The three URL flags take full endpoints and override only their matching service for that call. Expert–Spokesperson uses `--realtime-url` for both input and output; `--stt-url` and `--tts-url` apply only to conventional mode. Without an override, each service uses its OpenAI endpoint (or its existing environment override). `OPENAI_API_KEY` supplies the CLI credential; the desktop app stores keys separately by endpoint URL in Keychain.
+By default the Spokesperson reads the Expert's replies aloud in its own voice. With `--expert-voice tts`, the Expert speaks for itself: berd-call plays each reply through the configured TTS backend, taking turns with the Spokesperson's audio, and your speech interrupts either of them. Afterwards the Spokesperson gets a silent note of what you heard, so it can follow up without repeating it. A reply settles the handoffs it answers once you hear any of it. `--voice`, `--rate`, `status` and `settings --tts` describe and change the Expert's voice. The Spokesperson keeps its own, shown as `session.spokesperson`: choose it with `--spokesperson-voice NAME` and `--spokesperson-rate 0.25–1.5`, at `start` or live with `settings`, and the menu bar offers both voices. Saved settings keep the Spokesperson's voice for the next call whose replies have their own, but not `--expert-voice`, so pass that on each `start`.
+
+The three URL flags take full endpoints and override only their matching service for that call. Expert–Spokesperson uses `--realtime-url` for both input and output; `--stt-url` applies only to conventional mode, and `--tts-url` also applies to the Expert's own voice. Without an override, each service uses its OpenAI endpoint (or its existing environment override). `OPENAI_API_KEY` supplies the CLI credential; the desktop app stores keys separately by endpoint URL in Keychain.
 
 The default Siri backend still requires an exact installed voice name and
 language. Missing or unavailable Siri voice configuration and an unavailable
@@ -263,7 +270,9 @@ retries cleanup. This avoids reporting a failed install after the model has
 already been applied.
 
 `voices.list` returns `backend`, normalized `languageFilter`,
-`availableLanguages`, and exact voice records. `voices.download` returns the
+`availableLanguages`, and exact voice records. A voice is `installed` only when
+sirittsd reports it downloaded and its assets are on disk: sirittsd can claim
+a voice whose assets are missing, then silently speak it in another voice. `voices.download` returns the
 canonical voice, `installed: true`, and `availabilityWaitSeconds`; a missing
 exact catalog identity fails with `voice_not_found` before any native download
 request. macOS status and install results contain `supported`, `locale`,

src-tauri/crates/berd-call/native/siri_tts_bridge.m

@@ -997,10 +997,14 @@ static void BerdDownloadedVoices(
         NSString *name = [resource valueForKey:@"name"] ?: @"";
         if (!name.length || !language.length) continue;
         NSString *key = [NSString stringWithFormat:@"%@|%@", name, normalizedLanguage];
+        // sirittsd can report a voice as downloaded whose assets aren't on disk,
+        // and then speaks it in another voice.
+        NSString *content = [resource valueForKey:@"contentPath"];
         byKey[key] = @{
             @"name" : name,
             @"language" : language,
             @"sizeBytes" : [resource valueForKey:@"assetSize"] ?: @0,
+            @"content" : @(content.length && [NSFileManager.defaultManager fileExistsAtPath:content]),
         };
     }
     return [[byKey allValues] sortedArrayUsingComparator:
@@ -1142,7 +1146,9 @@ static BOOL BerdTriggerDownload(NSString *language, NSError **error) {
             NSString *key = [NSString stringWithFormat:@"%@|%@",
                 candidate[@"name"], normalizedLanguage];
             NSMutableDictionary *voice = [candidate mutableCopy];
-            voice[@"installed"] = installedSnapshot[key] ?: @NO;
+            [voice removeObjectForKey:@"content"];
+            voice[@"installed"] =
+                [installedSnapshot[key] boolValue] && [candidate[@"content"] boolValue] ? @YES : @NO;
             [result addObject:voice];
         }
         NSData *json = [NSJSONSerialization dataWithJSONObject:result options:0 error:&error];

src-tauri/crates/berd-call/src/cli_help.rs

@@ -58,25 +58,32 @@ const SPEAK_HELP: &str = r#"Speak through the active local voice call.
 
 Usage:
   berd-call speak [--port PORT] [--re CURSOR]
-    [--resolves HANDOFF_ID]... TEXT
+    [--resolves HANDOFF_ID]... [--voice NAME [--language BCP47]] TEXT
 
 Speech waits for admission. The session setting controls whether it also waits
 for delivery. Non-blocking sessions require --stream or --codex and emit speech_result
 rows only for interruptions, failures, or pending user input. Interrupted results include
-estimatedSpokenText, a best-effort prefix based on delivered audio."#;
+estimatedSpokenText, a best-effort prefix based on delivered audio. --voice speaks
+this text in another voice of the call's reply backend without changing its saved
+voice, so several agents can share one call in distinct voices."#;
 
 const SETTINGS_HELP: &str = r#"Update settings for the active local voice call.
 
 Usage:
   berd-call settings [--port PORT] --non-blocking true|false
   berd-call settings [--port PORT] --rate FLOAT
   berd-call settings [--port PORT] --voice NAME [--language BCP47]
+  berd-call settings [--port PORT] --spokesperson-voice NAME
+  berd-call settings [--port PORT] --spokesperson-rate FLOAT
   berd-call settings [--port PORT] --tts JSON
   berd-call settings [--port PORT] --input-during-tts allow|suppress
   berd-call settings [--port PORT] --muted true|false
   berd-call settings [--port PORT] --restart [session options]
 
-The setting applies to subsequent speak calls. Non-blocking delivery requires
+The setting applies to subsequent speak calls. In a call started with
+--expert-voice tts, --voice and --rate change the agents' voice, and
+--spokesperson-voice and --spokesperson-rate change the Spokesperson's (an
+OpenAI realtime voice, 0.25 to 1.5). Non-blocking delivery requires
 a call started with --stream or --codex. Successful delivery produces no notification."#;
 
 const STATUS_HELP: &str = r#"Inspect the active local voice call.
@@ -113,13 +120,19 @@ Usage:
 Each form also accepts:
     [--rate FLOAT]
     [--stt-backend macos|parakeet|openai] [--stt-model-dir PATH]
-    [--mode conventional|expert-spokesperson]
+    [--mode conventional|expert-spokesperson] [--expert-voice tts|spokesperson]
+    [--spokesperson-voice NAME] [--spokesperson-rate 0.25..1.5]
     [--realtime-url WS_URL] [--stt-url WS_URL] [--tts-url HTTP_URL]
 
 The host owns microphone capture, audio playback, transcript delivery, and
 agent integration. See PROTOCOL.md for the framed stdin, stdout, and PCM
 contracts. Endpoint URL flags override only the matching service for this call
-and default to OpenAI when no environment override is set."#;
+and default to OpenAI when no environment override is set.
+
+In expert-spokesperson mode, --expert-voice tts speaks the Expert's replies
+through the configured TTS backend, in its own voice beside the Spokesperson's,
+instead of having the Spokesperson read them aloud. --spokesperson-voice and
+--spokesperson-rate then choose the Spokesperson's own voice and speed."#;
 
 const SYNTHESIZE_HELP: &str = r#"Render text through a configured TTS backend into a new WAV file.
 

src-tauri/crates/berd-call/src/configured_tts.rs

@@ -1,3 +1,4 @@
+use std::collections::HashMap;
 use std::fmt;
 use std::path::PathBuf;
 use std::sync::{Arc, Mutex};
@@ -214,6 +215,33 @@ impl TtsConfiguration {
         }
     }
 
+    /// This configuration in another of its backend's voices.
+    fn with_voice(mut self, name: &str, locale: Option<&str>) -> Result<Self, String> {
+        match &mut self {
+            Self::Siri {
+                voice, language, ..
+            } => {
+                *voice = name.into();
+                if let Some(locale) = locale {
+                    *language = locale.into();
+                }
+            }
+            Self::OpenAi { voice, .. } | Self::Pocket { voice, .. } => {
+                if locale.is_some() {
+                    return Err("only Siri voices take a language".into());
+                }
+                *voice = name.into();
+            }
+        }
+        if let Self::OpenAi { voice, .. } = &self {
+            if !crate::openai_realtime_protocol::OPENAI_REALTIME_VOICE_IDS.contains(&voice.as_str())
+            {
+                return Err(format!("OpenAI has no voice named {voice}"));
+            }
+        }
+        Ok(self)
+    }
+
     fn build(&self) -> Result<Arc<dyn TtsBackend>, String> {
         validate_settings(&self.settings())?;
         match self {
@@ -291,6 +319,8 @@ struct ConfiguredTtsState {
     configuration: TtsConfiguration,
     backend: Arc<dyn TtsBackend>,
     snapshot: TtsConfigurationSnapshot,
+    /// Backends for other voices, built for single utterances at this revision.
+    voices: HashMap<(String, Option<String>), TtsConfigurationLease>,
 }
 
 pub struct ConfiguredTtsSlot {
@@ -364,6 +394,7 @@ impl ConfiguredTtsSlot {
                     revision: 1,
                     settings,
                 },
+                voices: HashMap::new(),
             }),
         }
     }
@@ -386,6 +417,46 @@ impl ConfiguredTtsSlot {
         })
     }
 
+    /// Whether `lease_voice` already has this voice built.
+    pub fn has_voice(&self, voice: &str, language: Option<&str>) -> bool {
+        self.inner.lock().is_ok_and(|state| {
+            state
+                .voices
+                .contains_key(&(voice.to_string(), language.map(str::to_string)))
+        })
+    }
+
+    /// The configured backend speaking one utterance in another of its voices,
+    /// keeping the other settings. Building it doesn't hold the slot lock.
+    pub fn lease_voice(
+        &self,
+        voice: &str,
+        language: Option<&str>,
+    ) -> Result<TtsConfigurationLease, String> {
+        let key = (voice.to_string(), language.map(str::to_string));
+        let poisoned = || "configured TTS lock was poisoned".to_string();
+        let (configuration, revision) = {
+            let state = self.inner.lock().map_err(|_| poisoned())?;
+            if let Some(lease) = state.voices.get(&key) {
+                return Ok(lease.clone());
+            }
+            (state.configuration.clone(), state.snapshot.revision)
+        };
+        let configuration = configuration.with_voice(voice, language)?;
+        let lease = TtsConfigurationLease {
+            backend: configuration.build()?,
+            snapshot: TtsConfigurationSnapshot {
+                revision,
+                settings: configuration.settings(),
+            },
+        };
+        let mut state = self.inner.lock().map_err(|_| poisoned())?;
+        if state.snapshot.revision == revision {
+            state.voices.insert(key, lease.clone());
+        }
+        Ok(lease)
+    }
+
     /// Builds and validates a replacement without holding the slot lock.
     pub fn prepare_replacement(
         &self,
@@ -475,6 +546,7 @@ impl ConfiguredTtsSlot {
                 })?;
         state.configuration = replacement.configuration;
         state.backend = replacement.backend;
+        state.voices.clear();
         state.snapshot = TtsConfigurationSnapshot {
             revision,
             settings: replacement.settings,
@@ -554,6 +626,38 @@ mod tests {
         }
     }
 
+    #[test]
+    fn one_utterance_can_use_another_voice_without_changing_the_slot() {
+        let settings = |voice: &str| TtsSettings::OpenAi {
+            model: "gpt-4o-mini-tts".into(),
+            voice: voice.into(),
+            rate: 1.25,
+        };
+        let slot = ConfiguredTtsSlot::new_for_test(settings("marin"));
+        let cedar = slot.lease_voice("cedar", None).unwrap();
+        assert_eq!(cedar.snapshot().settings, settings("cedar"));
+        assert!(Arc::ptr_eq(
+            cedar.backend(),
+            slot.lease_voice("cedar", None).unwrap().backend()
+        ));
+        assert_eq!(slot.snapshot().unwrap().settings, settings("marin"));
+        assert!(slot
+            .lease_voice("nobody", None)
+            .err()
+            .unwrap()
+            .contains("no voice named nobody"));
+        assert!(slot
+            .lease_voice("cedar", Some("en-US"))
+            .err()
+            .unwrap()
+            .contains("language"));
+        slot.replace_for_test(1, settings("ash")).unwrap();
+        assert!(!Arc::ptr_eq(
+            cedar.backend(),
+            slot.lease_voice("cedar", None).unwrap().backend()
+        ));
+    }
+
     #[test]
     fn snapshot_is_sanitized_and_revisioned() {
         let slot = ConfiguredTtsSlot::new(TtsConfiguration::openai(