From a200e60b04cf4171854a5d182e6c8cf0e9d11702 Mon Sep 17 00:00:00 2001 From: John Tennant Date: Mon, 19 Oct 2026 16:41:35 -0400 Subject: [PATCH 01/10] berd-call: let the Expert speak in its own voice in realtime calls With `--expert-voice tts`, an Expert-Spokesperson session builds the configured TTS backend beside the realtime connection. Each reply plays through that backend in the Spokesperson's playback slot instead of being read aloud by the realtime model, so the same turn gates and barge-in apply to both voices. Afterwards the realtime session gets a silent note of what the user heard, and the handoffs the reply answers settle once any of it was heard. `session.tts` and `set_tts_settings` now describe the Expert's voice in that mode, and every Expert-Spokesperson session reports the realtime model and voice as `session.spokesperson`. Co-Authored-By: Claude Opus 5.5 Signed-off-by: John Tennant --- src-tauri/crates/berd-call/PROTOCOL.md | 6 + src-tauri/crates/berd-call/README.md | 5 +- src-tauri/crates/berd-call/src/cli_help.rs | 8 +- .../crates/berd-call/src/host_session.rs | 1 + src-tauri/crates/berd-call/src/main.rs | 466 +++++++++++++++++- .../berd-call/src/openai_realtime_protocol.rs | 10 + src-tauri/crates/berd-call/src/protocol.rs | 17 +- .../crates/berd-call/src/saved_settings.rs | 2 +- .../berd-call/tests/session_protocol.rs | 187 ++++++- 9 files changed, 674 insertions(+), 28 deletions(-) diff --git a/src-tauri/crates/berd-call/PROTOCOL.md b/src-tauri/crates/berd-call/PROTOCOL.md index 1c8922022..c1655f70a 100644 --- a/src-tauri/crates/berd-call/PROTOCOL.md +++ b/src-tauri/crates/berd-call/PROTOCOL.md @@ -72,6 +72,12 @@ OpenAI snapshots contain `model`, `voice`, and `rate`; Siri contains `voice`, and `rate`. Credentials, endpoints, and bundle paths never appear on stdout. Detailed backend errors are diagnostics on stderr only; protocol rejection and fatal messages are sanitized at the stdout boundary. +In Expert-Spokesperson sessions, `session.spokesperson` holds the realtime +`model` and `voice`. `session.tts` describes the voice that speaks the Expert's +replies: the configured TTS backend with `--expert-voice tts`, otherwise the +Spokesperson's own voice, and `set_tts_settings` changes that voice. A reply in +the Expert's own voice reports the same `admitted`, `speech_completed`, +`speech_interrupted`, and `speech_failed` lifecycle as a conventional reply. `session.input_during_tts` is the authoritative effective assistant-input policy and has its own revision. Status-sound settings are per-update parameters, not session snapshot configuration. The runtime owns cue cadence and playback; it diff --git a/src-tauri/crates/berd-call/README.md b/src-tauri/crates/berd-call/README.md index 752a84bc1..36ce6b8a1 100644 --- a/src-tauri/crates/berd-call/README.md +++ b/src-tauri/crates/berd-call/README.md @@ -142,11 +142,14 @@ berd-call session --tts-backend pocket --model-dir /path/to/native-voice-v2 --vo berd-call session --stt-backend parakeet --stt-model-dir /path/to/parakeet berd-call session --stt-backend openai berd-call session --tts-backend openai --mode expert-spokesperson --realtime-url ws://127.0.0.1:18870/v1/realtime +berd-call session --voice Aaron --language en-US --mode expert-spokesperson --expert-voice tts berd-call session --tts-backend openai --tts-url https://proxy.example/v1/audio/speech berd-call session --stt-backend openai --stt-url wss://proxy.example/v1/realtime?intent=transcription ``` -The three URL flags take full endpoints and override only their matching service for that call. Expert–Spokesperson uses `--realtime-url` for both input and output; `--stt-url` and `--tts-url` apply only to conventional mode. Without an override, each service uses its OpenAI endpoint (or its existing environment override). `OPENAI_API_KEY` supplies the CLI credential; the desktop app stores keys separately by endpoint URL in Keychain. +By default the Spokesperson reads the Expert's replies aloud in its own voice. With `--expert-voice tts`, the Expert speaks for itself: berd-call plays each reply through the configured TTS backend, taking turns with the Spokesperson's audio, and your speech interrupts either of them. Afterwards the Spokesperson gets a silent note of what you heard, so it can follow up without repeating it. A reply settles the handoffs it answers once you hear any of it. Its rate applies to the Expert's voice, and the Spokesperson's speed is that rate limited to 0.25–1.5. `status` and `settings --tts` describe and change the Expert's voice; the Spokesperson's model and voice appear as `session.spokesperson`. Saved settings don't keep `--expert-voice`, so pass it on each `start`. + +The three URL flags take full endpoints and override only their matching service for that call. Expert–Spokesperson uses `--realtime-url` for both input and output; `--stt-url` applies only to conventional mode, and `--tts-url` also applies to the Expert's own voice. Without an override, each service uses its OpenAI endpoint (or its existing environment override). `OPENAI_API_KEY` supplies the CLI credential; the desktop app stores keys separately by endpoint URL in Keychain. The default Siri backend still requires an exact installed voice name and language. Missing or unavailable Siri voice configuration and an unavailable diff --git a/src-tauri/crates/berd-call/src/cli_help.rs b/src-tauri/crates/berd-call/src/cli_help.rs index cc5448b85..f4e0d906b 100644 --- a/src-tauri/crates/berd-call/src/cli_help.rs +++ b/src-tauri/crates/berd-call/src/cli_help.rs @@ -113,13 +113,17 @@ Usage: Each form also accepts: [--rate FLOAT] [--stt-backend macos|parakeet|openai] [--stt-model-dir PATH] - [--mode conventional|expert-spokesperson] + [--mode conventional|expert-spokesperson] [--expert-voice tts|spokesperson] [--realtime-url WS_URL] [--stt-url WS_URL] [--tts-url HTTP_URL] The host owns microphone capture, audio playback, transcript delivery, and agent integration. See PROTOCOL.md for the framed stdin, stdout, and PCM contracts. Endpoint URL flags override only the matching service for this call -and default to OpenAI when no environment override is set."#; +and default to OpenAI when no environment override is set. + +In expert-spokesperson mode, --expert-voice tts speaks the Expert's replies +through the configured TTS backend, in its own voice beside the Spokesperson's, +instead of having the Spokesperson read them aloud."#; const SYNTHESIZE_HELP: &str = r#"Render text through a configured TTS backend into a new WAV file. diff --git a/src-tauri/crates/berd-call/src/host_session.rs b/src-tauri/crates/berd-call/src/host_session.rs index e28968fcf..132615ff2 100644 --- a/src-tauri/crates/berd-call/src/host_session.rs +++ b/src-tauri/crates/berd-call/src/host_session.rs @@ -3283,6 +3283,7 @@ mod tests { fn test_snapshot() -> VoiceSessionSnapshot { VoiceSessionSnapshot { + spokesperson: None, tts: berd_call::TtsConfigurationSnapshot { revision: 1, settings: berd_call::TtsSettings::Siri { diff --git a/src-tauri/crates/berd-call/src/main.rs b/src-tauri/crates/berd-call/src/main.rs index 7346405ec..217e8565c 100644 --- a/src-tauri/crates/berd-call/src/main.rs +++ b/src-tauri/crates/berd-call/src/main.rs @@ -36,8 +36,8 @@ use berd_call::openai_spokesperson::{ }; use berd_call::protocol::{ CancelOutcome, DismissHandoffsOutcome, ExpertTurnOutcome, InputDuringTtsOutcome, - NotAdmittedReason, OutputReadyOutcome, SessionMessage, SessionRequest, TtsSettingsOutcome, - VoiceSessionSnapshot, + NotAdmittedReason, OutputReadyOutcome, SessionMessage, SessionRequest, + SpokespersonVoiceSnapshot, TtsSettingsOutcome, VoiceSessionSnapshot, }; use berd_call::realtime_audio_delivery::RealtimeAudioDelivery; use berd_call::realtime_host_lifecycle::RealtimeSessionLossAction; @@ -189,6 +189,9 @@ struct SessionConfig { stt: SttBackendConfig, mode: SessionMode, endpoints: SessionEndpointOverrides, + /// In Expert-Spokesperson mode, speak the Expert's replies with `tts` + /// rather than having the Spokesperson say them. + expert_tts: bool, } #[derive(Clone, Debug, Default, PartialEq)] @@ -2187,6 +2190,7 @@ fn run_session(config: SessionConfig, pcm_output_fd: RawFd) -> Result<(), String let session = VoiceSessionSnapshot { tts: slot.snapshot()?, input_during_tts: input_policy.snapshot()?, + spokesperson: None, }; tts_slot = Some(slot); input_during_tts_slot = Some(input_policy); @@ -2608,6 +2612,15 @@ enum LivePlaybackInput { type LivePlaybackResult = Result<(String, u64), (String, u64, String)>; +/// How an Expert reply spoken in its own voice ended. +enum ReplyOutcome { + Completed, + Interrupted { spoken_through_utf8: u64 }, + Failed(String), +} + +type ReplyResult = (String, u64, ReplyOutcome); + struct LivePlayback { response_id: String, speech_id: u64, @@ -2857,6 +2870,9 @@ struct LiveResponse { prepare_id: Option, speech_id: Option, expert_text: Option, + /// An Expert reply spoken in its own voice, with the handoffs it answers. + /// It has no provider response; its whole text is played at once. + reply: Option>, delivery: RealtimeAudioDelivery, pending_audio: VecDeque>, pending_frames: usize, @@ -2874,6 +2890,7 @@ impl LiveResponse { prepare_id, speech_id, expert_text: None, + reply: None, delivery: RealtimeAudioDelivery::default(), pending_audio: VecDeque::new(), pending_frames: 0, @@ -3128,6 +3145,12 @@ fn apply_spokesperson_startup_settings( | TtsBackendConfig::Siri { rate, .. } | TtsBackendConfig::Pocket { rate, .. } => *rate, }; + // A rate for the Expert's own voice need not suit the Spokesperson's. + let rate = if session.expert_tts { + rate.clamp(0.25, 1.5) + } else { + rate + }; if !(0.25..=1.5).contains(&rate) { return Err("Expert-Spokesperson rate must be between 0.25 and 1.5".into()); } @@ -3145,6 +3168,12 @@ fn run_expert_spokesperson_session( let audio_transport = Arc::new(unsafe { AudioPipeTransport::from_raw_fd(pcm_output_fd)? }); let (audio_control_tx, audio_control_rx) = mpsc::channel(); let (playback_tx, playback_rx) = mpsc::channel::(); + let (reply_tx, reply_rx) = mpsc::channel::(); + // The Expert's own voice, with `--expert-voice tts`. + let mut reply_tts: Option> = None; + let (tts_configuration_tx, tts_configuration_rx) = mpsc::channel::(); + let mut tts_update: Option = None; + let mut next_tts_update_attempt = 0_u64; let stdout = io::stdout(); let mut writer = BufWriter::new(stdout.lock()); let mut core = RealtimeExpertSpokespersonSession::new(0, "external"); @@ -3364,6 +3393,14 @@ fn run_expert_spokesperson_session( &mut cancelled_directives, &mut writer, )?; + drop_waiting_replies( + &preexisting_responses, + active_response_id.as_deref(), + &mut responses, + &mut core, + runtime.as_ref().expect("initialized runtime"), + &mut writer, + )?; } else { turn_gate.finish_user_speaking(); } @@ -3999,6 +4036,64 @@ fn run_expert_spokesperson_session( } } } + poll_tts_configuration_update( + Instant::now(), + &tts_configuration_rx, + reply_tts.as_deref(), + &mut tts_update, + &mut writer, + )?; + while let Ok((response_id, speech_id, outcome)) = reply_rx.try_recv() { + if take_matching_live_playback(&mut active, &response_id, speech_id).is_none() { + continue; + } + let Some(mut response) = responses.remove(&response_id) else { + continue; + }; + let id = response + .prepare_id + .expect("an Expert reply has a prepare ID"); + let text = response.expert_text.take().unwrap_or_default(); + let (message, heard) = match outcome { + ReplyOutcome::Completed => ( + SessionMessage::SpeechCompleted { id, speech_id }, + text.len(), + ), + ReplyOutcome::Interrupted { + spoken_through_utf8, + } => ( + SessionMessage::SpeechInterrupted { + id, + speech_id, + spoken_through_utf8, + }, + usize::try_from(spoken_through_utf8).unwrap_or(text.len()), + ), + ReplyOutcome::Failed(message) => ( + SessionMessage::SpeechFailed { + id, + speech_id, + message, + }, + 0, + ), + }; + let heard = (0..=heard.min(text.len())) + .rev() + .find(|&end| text.is_char_boundary(end)) + .unwrap_or_default(); + let interrupted = heard < text.len(); + if response.claim_speech_terminal() { + write_message(&mut writer, &message)?; + } + finish_expert_reply( + &mut core, + runtime.as_ref().expect("initialized runtime"), + &text[..heard], + interrupted, + response.reply.as_deref().unwrap_or_default(), + )?; + } if initialized && !turn_gate.input_blocks_output() { start_next_live_playback( &mut active, @@ -4008,6 +4103,8 @@ fn run_expert_spokesperson_session( &audio_transport, &audio_control_tx, &playback_tx, + reply_tts.as_deref(), + &reply_tx, &mut core, runtime.as_ref().expect("initialized runtime"), &mut writer, @@ -4030,6 +4127,9 @@ fn run_expert_spokesperson_session( &mut core, runtime.as_ref().expect("initialized runtime"), &mut directive_speeches, + reply_tts.is_some(), + &mut responses, + &mut waiting_responses, &mut next_speech_id, &mut writer, )?; @@ -4065,6 +4165,12 @@ fn run_expert_spokesperson_session( &mut writer, )?; } + reject_tts_configuration_update( + &mut tts_update, + reply_tts.as_deref(), + "session is shutting down", + &mut writer, + )?; if let Some(delivery) = core.flush_expert_events("Voice conversation ended") { write_message( &mut writer, @@ -4149,6 +4255,12 @@ fn run_expert_spokesperson_session( spokesperson_config.endpoint.clone_from(url); } apply_spokesperson_startup_settings(&config, &mut spokesperson_config)?; + if config.expert_tts { + reply_tts = Some(Arc::new(create_tts_slot( + &config.tts, + config.endpoints.tts.as_deref(), + )?)); + } let tts = berd_call::TtsConfigurationSnapshot { revision: 1, settings: TtsSettings::OpenAi { @@ -4159,8 +4271,15 @@ fn run_expert_spokesperson_session( }; let input_policy = InputDuringTtsSlot::new(input_during_tts); let snapshot = VoiceSessionSnapshot { - tts: tts.clone(), + tts: match &reply_tts { + Some(slot) => slot.snapshot()?, + None => tts.clone(), + }, input_during_tts: input_policy.snapshot()?, + spokesperson: Some(SpokespersonVoiceSnapshot { + model: spokesperson_config.model().into(), + voice: spokesperson_config.voice().into(), + }), }; let (created, events) = OpenAiSpokespersonRuntime::spawn_observed(spokesperson_config.clone())?; @@ -4238,6 +4357,9 @@ fn run_expert_spokesperson_session( &mut core, runtime.as_ref().expect("initialized runtime"), &mut directive_speeches, + reply_tts.is_some(), + &mut responses, + &mut waiting_responses, &mut next_speech_id, &mut writer, )?, @@ -4524,6 +4646,23 @@ fn run_expert_spokesperson_session( { cancel_live_playback(&mut active); CancelOutcome::Cancelled + } else if let Some(response_id) = responses + .iter() + .find(|(_, response)| { + response.reply.is_some() && response.prepare_id == Some(id) + }) + .map(|(response_id, _)| response_id.clone()) + { + waiting_responses.retain(|waiting| *waiting != response_id); + drop_waiting_replies( + &HashSet::from([response_id]), + None, + &mut responses, + &mut core, + runtime.as_ref().expect("initialized runtime"), + &mut writer, + )?; + CancelOutcome::Cancelled } else { CancelOutcome::Stale }; @@ -4640,6 +4779,41 @@ fn run_expert_spokesperson_session( &mut writer, )?; } + Input::Request(SessionRequest::SetTtsSettings { + id, + expected_revision, + settings, + }) if reply_tts.is_some() => { + let slot = Arc::clone(reply_tts.as_ref().expect("checked reply voice")); + if tts_update.is_some() { + write_message( + &mut writer, + &SessionMessage::TtsSettingsResult { + id, + outcome: TtsSettingsOutcome::Rejected, + snapshot: slot.snapshot()?, + message: Some("another TTS configuration update is in progress".into()), + }, + )?; + } else { + let attempt = next_tts_update_attempt; + next_tts_update_attempt += 1; + tts_update = Some(ActiveTtsConfigurationUpdate { + attempt, + id, + deadline: Instant::now() + TTS_CONFIGURATION_TIMEOUT, + }); + let sender = tts_configuration_tx.clone(); + thread::spawn(move || { + let result = slot.prepare_replacement(expected_revision, settings); + let _ = sender.send(TtsConfigurationEvent { + attempt, + id, + result, + }); + }); + } + } Input::Request(SessionRequest::SetTtsSettings { id, expected_revision, @@ -4692,6 +4866,133 @@ fn run_expert_spokesperson_session( Ok(()) } +/// Speaks an Expert reply with its own voice in the Spokesperson's playback slot. +#[allow(clippy::too_many_arguments)] +fn spawn_reply_playback( + response_id: String, + speech_id: u64, + prepare_id: Option, + text: String, + backend: Arc, + transport: Arc, + control: mpsc::Sender, + completed: mpsc::Sender, +) -> Result { + let active = Arc::new(AtomicBool::new(true)); + let output = Arc::new(RemotePcmAudioOutput::new( + speech_id, + backend.pcm_spec(), + transport, + Arc::clone(&active), + control, + )?); + let worker_output = Arc::clone(&output); + let worker_active = Arc::clone(&active); + let worker_response_id = response_id.clone(); + thread::spawn(move || { + let outcome = match play_tts(&text, backend.as_ref(), &worker_output, &worker_active) { + Ok((true, _)) => ReplyOutcome::Completed, + Ok((false, delivery)) => ReplyOutcome::Interrupted { + spoken_through_utf8: u64::try_from(estimated_spoken_through_utf8(&text, &delivery)) + .expect("speech text is bounded well below u64"), + }, + Err(failure) => ReplyOutcome::Failed(failure.message), + }; + let _ = completed.send((worker_response_id, speech_id, outcome)); + }); + // A reply's audio comes from its worker, not from provider deltas. + let (sender, _) = mpsc::sync_channel(1); + Ok(LivePlayback { + response_id, + speech_id, + prepare_id, + output, + active, + sender, + }) +} + +/// Ends Expert replies that never started playing, as interrupted before any was heard. +fn drop_waiting_replies( + response_ids: &HashSet, + active_response_id: Option<&str>, + responses: &mut HashMap, + core: &mut RealtimeExpertSpokespersonSession, + runtime: &OpenAiSpokespersonRuntime, + writer: &mut impl Write, +) -> Result<(), String> { + for response_id in response_ids { + if Some(response_id.as_str()) == active_response_id + || !responses + .get(response_id) + .is_some_and(|response| response.reply.is_some()) + { + continue; + } + let response = responses.remove(response_id).expect("reply exists"); + write_message( + writer, + &SessionMessage::SpeechInterrupted { + id: response + .prepare_id + .expect("an Expert reply has a prepare ID"), + speech_id: response.speech_id.expect("an Expert reply has a speech ID"), + spoken_through_utf8: 0, + }, + )?; + finish_expert_reply( + core, + runtime, + "", + true, + response.reply.as_deref().unwrap_or_default(), + )?; + } + Ok(()) +} + +/// Tells the Spokesperson what the user heard of an Expert reply, and settles +/// the handoffs it answers. +fn finish_expert_reply( + core: &mut RealtimeExpertSpokespersonSession, + runtime: &OpenAiSpokespersonRuntime, + heard: &str, + interrupted: bool, + handoff_ids: &[String], +) -> Result<(), String> { + let heard = heard.trim(); + // Handoffs settled elsewhere while the reply played need no settling here. + let unknown = core.unknown_handoff_ids(handoff_ids); + let open: Vec = handoff_ids + .iter() + .filter(|id| !unknown.contains(id)) + .cloned() + .collect(); + if heard.is_empty() { + core.release_handoffs(&open); + } else { + core.dismiss_handoffs(&open)?; + } + let message = match (heard.is_empty(), interrupted) { + (true, _) => "The user interrupted the Expert's spoken reply before hearing any of it.".into(), + (false, false) => format!("The Expert said this to the user in its own voice:\n{heard}"), + (false, true) => format!( + "The Expert said this to the user in its own voice before the user interrupted:\n{heard}" + ), + }; + let coordination = core.request_expert_message(RealtimeExpertMessage { + message, + mode: RealtimeExpertMessageMode::Context, + event_id: None, + directive_id: None, + resolved_handoff_ids: Vec::new(), + })?; + for event in coordination.events { + runtime.send(SpokespersonCommand::Provider(event))?; + } + Ok(()) +} + fn spawn_live_playback( response_id: String, speech_id: u64, @@ -4748,11 +5049,15 @@ fn spawn_live_playback( }) } +#[allow(clippy::too_many_arguments)] fn submit_expert_prepare( request: PendingExpertPrepare, core: &mut RealtimeExpertSpokespersonSession, runtime: &OpenAiSpokespersonRuntime, directive_speeches: &mut HashMap, + expert_tts: bool, + responses: &mut HashMap, + waiting_responses: &mut VecDeque, next_speech_id: &mut u64, writer: &mut impl Write, ) -> Result<(), String> { @@ -4790,26 +5095,39 @@ fn submit_expert_prepare( } => { let speech_id = *next_speech_id; *next_speech_id += 1; - directive_speeches.insert( - request.id, - DirectiveSpeech { - prepare_id: request.id, - speech_id, - text: message.clone(), - }, - ); core.record_expert_turn(message.clone()); let resolved_handoff_ids = request.resolved_handoff_ids; core.mark_handoffs_resolving(&resolved_handoff_ids)?; - let coordination = core.request_expert_message(RealtimeExpertMessage { - message, - mode: RealtimeExpertMessageMode::Say, - event_id: None, - directive_id: Some(request.id), - resolved_handoff_ids, - })?; - for event in coordination.events { - runtime.send(SpokespersonCommand::Provider(event))?; + if expert_tts { + // Played in turn with the Spokesperson's responses, without one of its own. + let response_id = format!("expert-reply-{speech_id}"); + let mut response = LiveResponse::new(Some(request.id), Some(speech_id)); + response.expert_text = Some(message); + response.reply = Some(resolved_handoff_ids); + response.audio_done = true; + response.finish_sent = true; + response.server_finished = true; + responses.insert(response_id.clone(), response); + waiting_responses.push_back(response_id); + } else { + directive_speeches.insert( + request.id, + DirectiveSpeech { + prepare_id: request.id, + speech_id, + text: message.clone(), + }, + ); + let coordination = core.request_expert_message(RealtimeExpertMessage { + message, + mode: RealtimeExpertMessageMode::Say, + event_id: None, + directive_id: Some(request.id), + resolved_handoff_ids, + })?; + for event in coordination.events { + runtime.send(SpokespersonCommand::Provider(event))?; + } } write_message( writer, @@ -4944,6 +5262,8 @@ fn start_next_live_playback( audio_transport: &Arc, audio_control_tx: &mpsc::Sender, playback_tx: &mpsc::Sender, + reply_tts: Option<&ConfiguredTtsSlot>, + reply_tx: &mpsc::Sender, core: &mut RealtimeExpertSpokespersonSession, runtime: &OpenAiSpokespersonRuntime, writer: &mut impl Write, @@ -4955,7 +5275,24 @@ fn start_next_live_playback( let Some(response) = responses.get_mut(&response_id) else { continue; }; - if response.interrupted || response.playback_complete || response.pending_audio.is_empty() { + if response.interrupted || response.playback_complete { + continue; + } + if response.reply.is_some() { + let tts = reply_tts.ok_or("an Expert reply has no voice of its own")?; + *active = Some(spawn_reply_playback( + response_id, + response.speech_id.expect("an Expert reply has a speech ID"), + response.prepare_id, + response.expert_text.clone().unwrap_or_default(), + Arc::clone(tts.lease()?.backend()), + Arc::clone(audio_transport), + audio_control_tx.clone(), + reply_tx.clone(), + )?); + break; + } + if response.pending_audio.is_empty() { continue; } let speech_id = response.speech_id.unwrap_or_else(|| { @@ -5401,6 +5738,7 @@ fn parse_args(args: &[String]) -> Result { let mut stt_model_dir = None; let mut mode = SessionMode::Conventional; let mut endpoints = SessionEndpointOverrides::default(); + let mut expert_tts = false; let mut index = 2; while index < args.len() { let flag = args[index].as_str(); @@ -5436,6 +5774,13 @@ fn parse_args(args: &[String]) -> Result { _ => return Err("--mode must be conventional or expert-spokesperson".into()), } } + "--expert-voice" => { + expert_tts = match value.as_str() { + "tts" => true, + "spokesperson" => false, + _ => return Err("--expert-voice must be tts or spokesperson".into()), + } + } "--pcm-output-fd" => {} _ => return Err(format!("unknown argument: {flag}").into()), } @@ -5446,8 +5791,11 @@ fn parse_args(args: &[String]) -> Result { if endpoints.realtime.is_some() && mode != SessionMode::ExpertSpokesperson { return Err("--realtime-url requires --mode expert-spokesperson".into()); } + if expert_tts && mode != SessionMode::ExpertSpokesperson { + return Err("--expert-voice requires --mode expert-spokesperson".into()); + } if mode == SessionMode::ExpertSpokesperson - && (endpoints.tts.is_some() || endpoints.stt.is_some()) + && (endpoints.stt.is_some() || endpoints.tts.is_some() && !expert_tts) { return Err("--tts-url and --stt-url apply only to conventional mode; Expert-Spokesperson uses --realtime-url".into()); } @@ -5462,6 +5810,7 @@ fn parse_args(args: &[String]) -> Result { stt, mode, endpoints, + expert_tts, }) } @@ -9110,6 +9459,7 @@ mod tests { session: VoiceSessionSnapshot { tts: snapshot.clone(), input_during_tts: test_input_policy(), + spokesperson: None, }, }) .unwrap(); @@ -9696,6 +10046,7 @@ mod tests { stt: SttBackendConfig::Macos, mode: SessionMode::Conventional, endpoints: SessionEndpointOverrides::default(), + expert_tts: false, } ); @@ -9706,6 +10057,7 @@ mod tests { stt: SttBackendConfig::Macos, mode: SessionMode::Conventional, endpoints: SessionEndpointOverrides::default(), + expert_tts: false, } ); } @@ -9782,6 +10134,7 @@ mod tests { stt: SttBackendConfig::Macos, mode: SessionMode::Conventional, endpoints: SessionEndpointOverrides::default(), + expert_tts: false, } ); assert!(parse_args(&args(&[ @@ -9837,6 +10190,7 @@ mod tests { stt: SttBackendConfig::Macos, mode: SessionMode::ExpertSpokesperson, endpoints: SessionEndpointOverrides::default(), + expert_tts: false, }; let mut realtime = OpenAiSpokespersonConfig { endpoint: "ws://localhost".into(), @@ -9857,6 +10211,71 @@ mod tests { assert_eq!(realtime.voice(), "marin"); } + #[test] + fn expert_voice_is_an_expert_spokesperson_choice_with_its_own_tts() { + let parse = |extra: &[&str]| { + let mut all = vec!["berd-call", "session"]; + all.extend_from_slice(extra); + parse_args(&args(&all)) + }; + let expert = |extra: &[&str]| { + let mut all = vec!["--mode", "expert-spokesperson"]; + all.extend_from_slice(extra); + parse(&all) + }; + assert!(!expert(&["--tts-backend", "openai"]).unwrap().expert_tts); + assert!( + !expert(&["--tts-backend", "openai", "--expert-voice", "spokesperson"]) + .unwrap() + .expert_tts + ); + let own = expert(&[ + "--tts-backend", + "openai", + "--rate", + "2.0", + "--expert-voice", + "tts", + "--tts-url", + "http://127.0.0.1:18870/v1/audio/speech", + ]) + .unwrap(); + assert!(own.expert_tts); + assert_eq!( + own.endpoints.tts.as_deref(), + Some("http://127.0.0.1:18870/v1/audio/speech") + ); + let mut realtime = OpenAiSpokespersonConfig { + endpoint: "ws://localhost".into(), + api_key: "test-key".into(), + session: berd_call::openai_realtime_protocol::RealtimeSpokespersonSessionOptions { + model: Some("test-model".into()), + voice: Some("marin".into()), + ..Default::default() + }, + semantic_transcript: Vec::new(), + }; + apply_spokesperson_startup_settings(&own, &mut realtime).unwrap(); + assert_eq!(realtime.speed(), 1.5); + let read_aloud = expert(&["--tts-backend", "openai", "--rate", "2.0"]).unwrap(); + assert!(apply_spokesperson_startup_settings(&read_aloud, &mut realtime).is_err()); + assert!(expert(&["--expert-voice", "loud"]) + .unwrap_err() + .contains("tts or spokesperson")); + assert!(parse(&["--tts-backend", "openai", "--expert-voice", "tts"]) + .unwrap_err() + .contains("--mode expert-spokesperson")); + assert_eq!( + saved_settings::persistable_arguments(&args(&[ + "--mode", + "expert-spokesperson", + "--expert-voice", + "tts" + ])), + args(&["--mode", "expert-spokesperson"]) + ); + } + #[test] fn session_endpoint_overrides_are_independent_and_require_matching_modes() { let realtime = parse_args(&args(&[ @@ -9965,6 +10384,7 @@ mod tests { stt: SttBackendConfig::Macos, mode: SessionMode::Conventional, endpoints: SessionEndpointOverrides::default(), + expert_tts: false, } ); assert!(parse_args(&args(&[ @@ -10026,6 +10446,7 @@ mod tests { }, mode: SessionMode::Conventional, endpoints: SessionEndpointOverrides::default(), + expert_tts: false, } ); assert!(parse_args(&args(&[ @@ -10470,6 +10891,7 @@ mod tests { stt: SttBackendConfig::OpenAi, mode: SessionMode::Conventional, endpoints: SessionEndpointOverrides::default(), + expert_tts: false, } ); } diff --git a/src-tauri/crates/berd-call/src/openai_realtime_protocol.rs b/src-tauri/crates/berd-call/src/openai_realtime_protocol.rs index 2bc4afc6f..570299a09 100644 --- a/src-tauri/crates/berd-call/src/openai_realtime_protocol.rs +++ b/src-tauri/crates/berd-call/src/openai_realtime_protocol.rs @@ -1783,6 +1783,16 @@ impl RealtimeExpertSpokespersonSession { Ok(()) } + /// Leaves handoffs open for another reply once the one meant to answer + /// them went unheard. + pub fn release_handoffs(&mut self, handoff_ids: &[String]) { + for handoff_id in handoff_ids { + if let Some(handoff) = self.open_handoffs.get_mut(handoff_id) { + handoff.resolving = false; + } + } + } + pub fn dismiss_handoffs(&mut self, handoff_ids: &[String]) -> Result<(), String> { let unknown = self.unknown_handoff_ids(handoff_ids); if !unknown.is_empty() { diff --git a/src-tauri/crates/berd-call/src/protocol.rs b/src-tauri/crates/berd-call/src/protocol.rs index 32eca4e91..e27904bd0 100644 --- a/src-tauri/crates/berd-call/src/protocol.rs +++ b/src-tauri/crates/berd-call/src/protocol.rs @@ -157,8 +157,19 @@ pub enum OutputReadyOutcome { #[derive(Clone, Debug, Deserialize, Serialize, PartialEq)] pub struct VoiceSessionSnapshot { + /// The voice that speaks replies: the conventional backend, or the + /// Spokesperson's own voice when it reads them. pub tts: TtsConfigurationSnapshot, pub input_during_tts: InputDuringTtsSnapshot, + /// The Spokesperson's voice, in Expert-Spokesperson sessions. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub spokesperson: Option, +} + +#[derive(Clone, Debug, Deserialize, Serialize, PartialEq, Eq)] +pub struct SpokespersonVoiceSnapshot { + pub model: String, + pub voice: String, } #[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq, Eq)] @@ -402,10 +413,14 @@ mod tests { revision: 1, policy: InputDuringTtsPolicy::AllowBargeIn, }, + spokesperson: Some(SpokespersonVoiceSnapshot { + model: "gpt-realtime".into(), + voice: "cedar".into(), + }), }, }) .unwrap(), - r#"{"type":"ready","id":4,"protocol":5,"session":{"tts":{"revision":1,"backend":"openai","model":"gpt-4o-mini-tts","voice":"marin","rate":1.0},"input_during_tts":{"revision":1,"policy":"allow_barge_in"}}}"# + r#"{"type":"ready","id":4,"protocol":5,"session":{"tts":{"revision":1,"backend":"openai","model":"gpt-4o-mini-tts","voice":"marin","rate":1.0},"input_during_tts":{"revision":1,"policy":"allow_barge_in"},"spokesperson":{"model":"gpt-realtime","voice":"cedar"}}}"# ); assert_eq!( serde_json::from_str::( diff --git a/src-tauri/crates/berd-call/src/saved_settings.rs b/src-tauri/crates/berd-call/src/saved_settings.rs index 8587b52b1..0caf56779 100644 --- a/src-tauri/crates/berd-call/src/saved_settings.rs +++ b/src-tauri/crates/berd-call/src/saved_settings.rs @@ -131,7 +131,7 @@ pub(crate) fn persistable_arguments(arguments: &[String]) -> Vec { .filter(|pair| { !matches!( pair[0].as_str(), - "--realtime-url" | "--stt-url" | "--tts-url" + "--realtime-url" | "--stt-url" | "--tts-url" | "--expert-voice" ) }) .flatten() diff --git a/src-tauri/crates/berd-call/tests/session_protocol.rs b/src-tauri/crates/berd-call/tests/session_protocol.rs index 89c8339a3..cd9ea0ca6 100644 --- a/src-tauri/crates/berd-call/tests/session_protocol.rs +++ b/src-tauri/crates/berd-call/tests/session_protocol.rs @@ -32,6 +32,7 @@ struct ExpertSpokespersonTestSession { output: mpsc::Receiver, stderr: mpsc::Receiver, audio_host: Option>, + ready: Value, } struct AudioCancellationGate { @@ -54,6 +55,28 @@ impl ExpertSpokespersonTestSession { played_frame_limit: Option, played_ready: Option>, cancellation_gate: Option, + ) -> Self { + Self::launch( + endpoint, + renew_after_ms, + played_frame_limit, + played_ready, + cancellation_gate, + &[], + ) + } + + fn start_with_args(endpoint: String, args: &[&str]) -> Self { + Self::launch(endpoint, None, None, None, None, args) + } + + fn launch( + endpoint: String, + renew_after_ms: Option, + played_frame_limit: Option, + played_ready: Option>, + cancellation_gate: Option, + args: &[&str], ) -> Self { let (mut command, _pcm, audio_host) = session_command(); if let Some(renew_after_ms) = renew_after_ms { @@ -72,6 +95,7 @@ impl ExpertSpokespersonTestSession { "--realtime-url", &endpoint, ]) + .args(args) .env("OPENAI_API_KEY", "test-key") .env("OPENAI_REALTIME_ENDPOINT", "ws://127.0.0.1:1/unused") .env("OPENAI_REALTIME_MODEL", "test-model") @@ -99,11 +123,13 @@ impl ExpertSpokespersonTestSession { output, stderr, audio_host: Some(audio_host), + ready: Value::Null, }; session.send(json!({ "type":"hello","id":1,"input_during_tts":"allow_barge_in" })); - assert_eq!(session.recv(Duration::from_secs(2))["type"], "ready"); + session.ready = session.recv(Duration::from_secs(2)); + assert_eq!(session.ready["type"], "ready"); session } @@ -2654,3 +2680,162 @@ fn siri_remote_output_supports_consecutive_turns_and_cancellation() { reader.join().unwrap(); audio_host.join().unwrap(); } + +/// Serves OpenAI speech requests with a little silence, reporting each request body. +fn spawn_speech_server() -> (String, mpsc::Receiver) { + let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap(); + let url = format!("http://{}/v1/audio/speech", listener.local_addr().unwrap()); + let (sender, receiver) = mpsc::channel(); + std::thread::spawn(move || { + for stream in listener.incoming() { + let mut stream = stream.unwrap(); + let mut request = Vec::new(); + let mut byte = [0_u8; 1]; + while !request.ends_with(b"\r\n\r\n") { + stream.read_exact(&mut byte).unwrap(); + request.push(byte[0]); + } + let head = String::from_utf8(request).unwrap().to_ascii_lowercase(); + let length = head + .lines() + .find_map(|line| line.strip_prefix("content-length:")) + .unwrap() + .trim() + .parse() + .unwrap(); + let mut body = vec![0_u8; length]; + stream.read_exact(&mut body).unwrap(); + let _ = sender.send(serde_json::from_slice(&body).unwrap()); + let audio = vec![1_u8; 4_800]; + write!( + stream, + "HTTP/1.1 200 OK\r\ncontent-type: audio/pcm\r\ncontent-length: {}\r\nconnection: close\r\n\r\n", + audio.len() + ) + .unwrap(); + stream.write_all(&audio).unwrap(); + } + }); + (url, receiver) +} + +#[test] +fn expert_voice_speaks_replies_itself_and_tells_the_spokesperson_what_was_heard() { + let (speech_url, speech_requests) = spawn_speech_server(); + let (endpoint_tx, endpoint_rx) = mpsc::sync_channel(1); + let (context_tx, context_rx) = mpsc::sync_channel(1); + let server = std::thread::spawn(move || { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap() + .block_on(async move { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + endpoint_tx + .send(format!("ws://{}/", listener.local_addr().unwrap())) + .unwrap(); + let (stream, _) = listener.accept().await.unwrap(); + let mut socket = accept_async(stream).await.unwrap(); + let initial = receive_realtime_json(&mut socket).await; + acknowledge_realtime_session(&mut socket, &initial).await; + send_realtime_json( + &mut socket, + json!({ + "type":"response.function_call_arguments.done", + "response_id":"response-handoff", + "call_id":"handoff-reply", + "name":"handoff", + "arguments":"{\"message\":\"inspect the repository\"}" + }), + ) + .await; + let mut mentions = Vec::new(); + let context = loop { + let next = receive_realtime_json(&mut socket).await; + let text = next.to_string(); + if text.contains("in its own voice") { + break next; + } + if text.contains("I checked the repository.") { + mentions.push(next); + } + }; + let after = tokio::time::timeout(Duration::from_millis(300), async { + loop { + let next = receive_realtime_json(&mut socket).await; + if next["type"] == "response.create" { + break next; + } + } + }) + .await + .ok(); + context_tx.send((context, mentions, after)).unwrap(); + while socket.next().await.is_some() {} + }); + }); + let endpoint = endpoint_rx.recv().unwrap(); + let mut session = ExpertSpokespersonTestSession::start_with_args( + endpoint, + &["--expert-voice", "tts", "--tts-url", &speech_url], + ); + assert_eq!(session.ready["session"]["tts"]["backend"], "openai"); + assert_eq!( + session.ready["session"]["spokesperson"], + json!({"model":"test-model","voice":"old-voice"}) + ); + assert_eq!(session.recv(Duration::from_secs(2))["origin"], "handoff"); + assert_eq!( + session.recv(Duration::from_secs(2))["handoff_ids"], + json!(["handoff-external-1"]) + ); + + session.send(json!({ + "type":"prepare_speak", + "id":2, + "acknowledgement":1, + "text":"I checked the repository.", + "resolved_handoff_ids":["handoff-external-1"] + })); + let admitted = session.recv(Duration::from_secs(2)); + assert_eq!(admitted["type"], "admitted", "{admitted}"); + session.send(json!({ + "type":"output_ready", + "id":2, + "speech_id":admitted["speech_id"] + })); + let deadline = std::time::Instant::now() + Duration::from_secs(3); + loop { + let remaining = deadline + .checked_duration_since(std::time::Instant::now()) + .expect("the Expert's reply did not complete"); + let message = session.recv(remaining); + assert_ne!(message["type"], "fatal", "{message}"); + assert_ne!(message["type"], "speech_failed", "{message}"); + if message["type"] == "speech_completed" { + assert_eq!(message["id"], 2); + assert_eq!(message["speech_id"], admitted["speech_id"]); + break; + } + } + let speech = speech_requests + .recv_timeout(Duration::from_secs(1)) + .unwrap(); + assert_eq!(speech["input"], "I checked the repository."); + + let (context, mentions, after) = context_rx.recv_timeout(Duration::from_secs(2)).unwrap(); + assert_eq!(context["type"], "conversation.item.create", "{context}"); + assert!(context.to_string().contains( + "The Expert said this to the user in its own voice:\\nI checked the repository." + )); + assert!( + mentions.is_empty(), + "the Spokesperson was asked to say it: {mentions:?}" + ); + assert!( + after.is_none(), + "the context note started a response: {after:?}" + ); + session.shutdown(); + server.join().unwrap(); +} From c754eacad5e3a549b1cbe006b27ed7659858f03a Mon Sep 17 00:00:00 2001 From: John Tennant Date: Mon, 19 Oct 2026 17:05:00 -0400 Subject: [PATCH 02/10] berd-call: give the Spokesperson its own voice controls beside the Expert's With --expert-voice tts, the one voice and speed control changed the Expert's voice, leaving no way to change the Spokesperson's during a call. The Spokesperson now has its own controls in these calls: - `set_spokesperson_settings` swaps in a replacement realtime session, the same way `set_tts_settings` changes the Spokesperson's voice elsewhere. - `settings --spokesperson-voice` and `--spokesperson-rate`, plus matching `start` options, set its voice and speed. - The menu bar shows Agent and Spokesperson voice and speed menus side by side, each limited to its own range of speeds. `session.spokesperson` is now the Spokesperson's full revisioned OpenAI snapshot. It appears only in calls whose replies have their own voice, since elsewhere `session.tts` is already the Spokesperson's. The Spokesperson's speed no longer follows the Expert's rate. The voice and speed it was last given are saved, and come back in the next call whose replies have their own voice. Co-Authored-By: Claude Opus 5.5 Signed-off-by: John Tennant --- src-tauri/crates/berd-call/PROTOCOL.md | 11 +- src-tauri/crates/berd-call/README.md | 2 +- src-tauri/crates/berd-call/src/cli_help.rs | 11 +- .../crates/berd-call/src/host_control.rs | 17 ++ .../crates/berd-call/src/host_session.rs | 168 ++++++++++++- src-tauri/crates/berd-call/src/main.rs | 224 ++++++++++++++++-- src-tauri/crates/berd-call/src/menu_bar.rs | 153 +++++++++--- src-tauri/crates/berd-call/src/protocol.rs | 28 ++- .../crates/berd-call/src/saved_settings.rs | 28 +++ .../berd-call/tests/session_protocol.rs | 69 +++++- 10 files changed, 629 insertions(+), 82 deletions(-) diff --git a/src-tauri/crates/berd-call/PROTOCOL.md b/src-tauri/crates/berd-call/PROTOCOL.md index c1655f70a..0d1d20639 100644 --- a/src-tauri/crates/berd-call/PROTOCOL.md +++ b/src-tauri/crates/berd-call/PROTOCOL.md @@ -72,10 +72,12 @@ OpenAI snapshots contain `model`, `voice`, and `rate`; Siri contains `voice`, and `rate`. Credentials, endpoints, and bundle paths never appear on stdout. Detailed backend errors are diagnostics on stderr only; protocol rejection and fatal messages are sanitized at the stdout boundary. -In Expert-Spokesperson sessions, `session.spokesperson` holds the realtime -`model` and `voice`. `session.tts` describes the voice that speaks the Expert's -replies: the configured TTS backend with `--expert-voice tts`, otherwise the -Spokesperson's own voice, and `set_tts_settings` changes that voice. A reply in +`session.tts` describes the voice that speaks replies, and `set_tts_settings` +changes it: in Expert-Spokesperson sessions, that is the configured TTS backend +with `--expert-voice tts` and otherwise the Spokesperson's own voice. With +`--expert-voice tts`, `session.spokesperson` is the Spokesperson's own OpenAI +snapshot, with its own revision, and `set_spokesperson_settings` changes it the +way `set_tts_settings` changes the Spokesperson's voice in other sessions. A reply in the Expert's own voice reports the same `admitted`, `speech_completed`, `speech_interrupted`, and `speech_failed` lifecycle as a conventional reply. `session.input_during_tts` is the authoritative effective assistant-input @@ -201,6 +203,7 @@ it never admits a replacement while old host audio may still be active. {"type":"set_input_muted","id":u64,"active":bool} {"type":"set_conversation_status","id":u64,"status":"working"|"waiting","settings":StatusSoundSettings} {"type":"set_tts_settings","id":u64,"expected_revision":u64,"settings":TtsSettings} +{"type":"set_spokesperson_settings","id":u64,"expected_revision":u64,"settings":TtsSettings} {"type":"set_input_during_tts","id":u64,"expected_revision":u64,"policy":"allow_barge_in"|"suppress_input"} {"type":"reset_input","id":u64} {"type":"prepare_speak","id":u64,"acknowledgement":u64|null,"text":string} diff --git a/src-tauri/crates/berd-call/README.md b/src-tauri/crates/berd-call/README.md index 36ce6b8a1..d48af1818 100644 --- a/src-tauri/crates/berd-call/README.md +++ b/src-tauri/crates/berd-call/README.md @@ -147,7 +147,7 @@ berd-call session --tts-backend openai --tts-url https://proxy.example/v1/audio/ berd-call session --stt-backend openai --stt-url wss://proxy.example/v1/realtime?intent=transcription ``` -By default the Spokesperson reads the Expert's replies aloud in its own voice. With `--expert-voice tts`, the Expert speaks for itself: berd-call plays each reply through the configured TTS backend, taking turns with the Spokesperson's audio, and your speech interrupts either of them. Afterwards the Spokesperson gets a silent note of what you heard, so it can follow up without repeating it. A reply settles the handoffs it answers once you hear any of it. Its rate applies to the Expert's voice, and the Spokesperson's speed is that rate limited to 0.25–1.5. `status` and `settings --tts` describe and change the Expert's voice; the Spokesperson's model and voice appear as `session.spokesperson`. Saved settings don't keep `--expert-voice`, so pass it on each `start`. +By default the Spokesperson reads the Expert's replies aloud in its own voice. With `--expert-voice tts`, the Expert speaks for itself: berd-call plays each reply through the configured TTS backend, taking turns with the Spokesperson's audio, and your speech interrupts either of them. Afterwards the Spokesperson gets a silent note of what you heard, so it can follow up without repeating it. A reply settles the handoffs it answers once you hear any of it. `--voice`, `--rate`, `status` and `settings --tts` describe and change the Expert's voice. The Spokesperson keeps its own, shown as `session.spokesperson`: choose it with `--spokesperson-voice NAME` and `--spokesperson-rate 0.25–1.5`, at `start` or live with `settings`, and the menu bar offers both voices. Saved settings keep the Spokesperson's voice for the next call whose replies have their own, but not `--expert-voice`, so pass that on each `start`. The three URL flags take full endpoints and override only their matching service for that call. Expert–Spokesperson uses `--realtime-url` for both input and output; `--stt-url` applies only to conventional mode, and `--tts-url` also applies to the Expert's own voice. Without an override, each service uses its OpenAI endpoint (or its existing environment override). `OPENAI_API_KEY` supplies the CLI credential; the desktop app stores keys separately by endpoint URL in Keychain. diff --git a/src-tauri/crates/berd-call/src/cli_help.rs b/src-tauri/crates/berd-call/src/cli_help.rs index f4e0d906b..4d5edab41 100644 --- a/src-tauri/crates/berd-call/src/cli_help.rs +++ b/src-tauri/crates/berd-call/src/cli_help.rs @@ -71,12 +71,17 @@ Usage: berd-call settings [--port PORT] --non-blocking true|false berd-call settings [--port PORT] --rate FLOAT berd-call settings [--port PORT] --voice NAME [--language BCP47] + berd-call settings [--port PORT] --spokesperson-voice NAME + berd-call settings [--port PORT] --spokesperson-rate FLOAT berd-call settings [--port PORT] --tts JSON berd-call settings [--port PORT] --input-during-tts allow|suppress berd-call settings [--port PORT] --muted true|false berd-call settings [--port PORT] --restart [session options] -The setting applies to subsequent speak calls. Non-blocking delivery requires +The setting applies to subsequent speak calls. In a call started with +--expert-voice tts, --voice and --rate change the agents' voice, and +--spokesperson-voice and --spokesperson-rate change the Spokesperson's (an +OpenAI realtime voice, 0.25 to 1.5). Non-blocking delivery requires a call started with --stream or --codex. Successful delivery produces no notification."#; const STATUS_HELP: &str = r#"Inspect the active local voice call. @@ -114,6 +119,7 @@ Each form also accepts: [--rate FLOAT] [--stt-backend macos|parakeet|openai] [--stt-model-dir PATH] [--mode conventional|expert-spokesperson] [--expert-voice tts|spokesperson] + [--spokesperson-voice NAME] [--spokesperson-rate 0.25..1.5] [--realtime-url WS_URL] [--stt-url WS_URL] [--tts-url HTTP_URL] The host owns microphone capture, audio playback, transcript delivery, and @@ -123,7 +129,8 @@ and default to OpenAI when no environment override is set. In expert-spokesperson mode, --expert-voice tts speaks the Expert's replies through the configured TTS backend, in its own voice beside the Spokesperson's, -instead of having the Spokesperson read them aloud."#; +instead of having the Spokesperson read them aloud. --spokesperson-voice and +--spokesperson-rate then choose the Spokesperson's own voice and speed."#; const SYNTHESIZE_HELP: &str = r#"Render text through a configured TTS backend into a new WAV file. diff --git a/src-tauri/crates/berd-call/src/host_control.rs b/src-tauri/crates/berd-call/src/host_control.rs index f3d56d714..cd00eb516 100644 --- a/src-tauri/crates/berd-call/src/host_control.rs +++ b/src-tauri/crates/berd-call/src/host_control.rs @@ -34,6 +34,7 @@ pub(crate) trait HostControl: Send + Sync + 'static { fn set_tts(&self, settings: berd_call::TtsSettings) -> Result; fn set_rate(&self, rate: f32) -> Result; fn set_voice(&self, voice: String, language: Option) -> Result; + fn set_spokesperson(&self, voice: Option, rate: Option) -> Result; fn set_input_during_tts( &self, policy: berd_call::input::InputDuringTtsPolicy, @@ -129,6 +130,11 @@ pub(crate) enum ControlRequest { voice: String, language: Option, }, + /// The Spokesperson's voice or speed, when replies have their own. + Spokesperson { + voice: Option, + rate: Option, + }, InputDuringTts { policy: berd_call::input::InputDuringTtsPolicy, }, @@ -187,6 +193,7 @@ pub(crate) fn begin_request(port: u16, request: ControlRequest) -> Result Result ControlRequest::TtsSettings { settings } => control.set_tts(settings), ControlRequest::Rate { rate } => control.set_rate(rate), ControlRequest::Voice { voice, language } => control.set_voice(voice, language), + ControlRequest::Spokesperson { voice, rate } => { + control.set_spokesperson(voice, rate) + } ControlRequest::NonBlocking { enabled } => control.set_non_blocking(enabled), ControlRequest::InputDuringTts { policy } => control.set_input_during_tts(policy), ControlRequest::Muted { muted } => control.set_muted(muted), @@ -349,6 +359,13 @@ mod tests { fn set_voice(&self, voice: String, language: Option) -> Result { Ok(json!({"voice": voice, "language": language})) } + fn set_spokesperson( + &self, + voice: Option, + rate: Option, + ) -> Result { + Ok(json!({"spokesperson": {"voice": voice, "rate": rate}})) + } fn set_non_blocking(&self, enabled: bool) -> Result { Ok(json!({"nonBlocking": enabled})) } diff --git a/src-tauri/crates/berd-call/src/host_session.rs b/src-tauri/crates/berd-call/src/host_session.rs index 132615ff2..ac01d0fa5 100644 --- a/src-tauri/crates/berd-call/src/host_session.rs +++ b/src-tauri/crates/berd-call/src/host_session.rs @@ -508,6 +508,11 @@ enum ControlCommand { expected_revision: u64, response: SyncSender>, }, + SpokespersonSettings { + settings: berd_call::TtsSettings, + expected_revision: u64, + response: SyncSender>, + }, Speak { text: String, acknowledgement: Option, @@ -644,6 +649,38 @@ impl HostControl for SessionControl { }) } + fn set_spokesperson(&self, voice: Option, rate: Option) -> Result { + let (mut settings, expected_revision) = { + let session = self + .ready + .session + .lock() + .map_err(|_| "session settings lock failed")?; + let spokesperson = session + .spokesperson + .as_ref() + .ok_or("this call has no separate Spokesperson voice; use --voice and --rate")?; + (spokesperson.settings.clone(), spokesperson.revision) + }; + if let ( + Some(voice), + berd_call::TtsSettings::OpenAi { + voice: selected, .. + }, + ) = (voice, &mut settings) + { + *selected = voice; + } + if let Some(rate) = rate { + settings = settings.with_rate(rate); + } + self.request(|response| ControlCommand::SpokespersonSettings { + settings, + expected_revision, + response, + }) + } + fn set_input_during_tts(&self, policy: InputDuringTtsPolicy) -> Result { let expected_revision = self .ready @@ -752,6 +789,7 @@ enum AudioCommand { enum SavedPreference { SessionStart, Tts, + Spokesperson, InputPolicy, } @@ -767,6 +805,8 @@ struct SessionActor { next_id: u64, pending_speak: Option, pending_settings: std::collections::HashMap>>, + // Settings requests answered with the Spokesperson's voice rather than the call's. + spokesperson_settings: HashSet, waiting_speaks: VecDeque, session: Arc>, transcript: Arc, @@ -831,6 +871,7 @@ impl SessionActor { next_id: 2, pending_speak: None, pending_settings: std::collections::HashMap::new(), + spokesperson_settings: HashSet::new(), waiting_speaks: VecDeque::new(), session, transcript, @@ -1030,6 +1071,23 @@ impl SessionActor { )?; self.pending_settings.insert(id, response); } + ControlCommand::SpokespersonSettings { + settings, + expected_revision, + response, + } => { + let id = self.next_id(); + send_request( + &self.writer, + &SessionRequest::SetSpokespersonSettings { + id, + expected_revision, + settings, + }, + )?; + self.spokesperson_settings.insert(id); + self.pending_settings.insert(id, response); + } ControlCommand::Speak { text, acknowledgement, @@ -1151,13 +1209,23 @@ impl SessionActor { snapshot, message, } => { + let spokesperson = self.spokesperson_settings.remove(&id); if let Ok(mut session) = self.session.lock() { - if snapshot.revision >= session.tts.revision { - session.tts = snapshot.clone(); + let voice = if spokesperson { + session.spokesperson.as_mut() + } else { + Some(&mut session.tts) + }; + if let Some(voice) = voice.filter(|voice| snapshot.revision >= voice.revision) { + *voice = snapshot.clone(); } } if outcome == berd_call::protocol::TtsSettingsOutcome::Applied { - self.persist_settings(SavedPreference::Tts); + self.persist_settings(if spokesperson { + SavedPreference::Spokesperson + } else { + SavedPreference::Tts + }); } if let Some(response) = self.pending_settings.remove(&id) { let _ = response.send(Ok( @@ -1388,6 +1456,26 @@ impl SessionActor { let Ok(session) = self.session.lock() else { return; }; + if let SavedPreference::Spokesperson = changed { + let Some(spokesperson) = &session.spokesperson else { + return; + }; + let replacement = vec![ + "--spokesperson-voice".to_string(), + spokesperson.settings.voice().to_string(), + "--spokesperson-rate".to_string(), + spokesperson.settings.rate().to_string(), + ]; + saved.arguments = + crate::saved_settings::merge_arguments(&saved.arguments, &replacement); + if let Err(error) = crate::saved_settings::update(path, |latest| { + latest.arguments = + crate::saved_settings::merge_arguments(&latest.arguments, &replacement) + }) { + eprintln!("berd-call: live settings applied but could not be saved: {error}"); + } + return; + } saved.tts = Some(session.tts.settings.clone()); let settings = &session.tts.settings; let pocket_model_dir = saved @@ -1435,6 +1523,7 @@ impl SessionActor { latest.tts = saved.tts.clone(); } SavedPreference::InputPolicy => latest.input_policy = saved.input_policy, + SavedPreference::Spokesperson => unreachable!("saved above"), }) { eprintln!("berd-call: live settings applied but could not be saved: {error}"); } @@ -3685,6 +3774,79 @@ mod tests { second_child.wait().unwrap(); } + #[test] + fn spokesperson_results_change_and_save_only_its_voice() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("settings.json"); + let mut child = Command::new("/bin/cat") + .stdin(Stdio::piped()) + .stdout(Stdio::null()) + .spawn() + .unwrap(); + let writer = Arc::new(Mutex::new(child.stdin.take().unwrap())); + let (_events, receiver) = mpsc::sync_channel(1); + let (_commands, commands) = mpsc::sync_channel(1); + let (audio, _audio_rx) = mpsc::sync_channel(1); + let mut actor = test_actor(writer, receiver, commands, audio); + let openai = |voice: &str, revision| berd_call::TtsConfigurationSnapshot { + revision, + settings: berd_call::TtsSettings::OpenAi { + model: "gpt-realtime".into(), + voice: voice.into(), + rate: 1.25, + }, + }; + actor.session.lock().unwrap().spokesperson = Some(openai("marin", 1)); + actor.saved = Some(( + path.clone(), + crate::saved_settings::SavedSettings { + arguments: ["--voice", "Aaron", "--language", "en-US"] + .map(str::to_string) + .to_vec(), + ..crate::saved_settings::SavedSettings::default() + }, + )); + let (response, result) = mpsc::sync_channel(1); + actor + .handle_command(ControlCommand::SpokespersonSettings { + settings: openai("cedar", 1).settings, + expected_revision: 1, + response, + }) + .unwrap(); + let id = *actor.spokesperson_settings.iter().next().unwrap(); + actor + .handle_event(SessionMessage::TtsSettingsResult { + id, + outcome: berd_call::protocol::TtsSettingsOutcome::Applied, + snapshot: openai("cedar", 2), + message: None, + }) + .unwrap(); + assert_eq!(result.recv().unwrap().unwrap()["outcome"], "applied"); + let session = actor.session.lock().unwrap().clone(); + assert_eq!(session.spokesperson, Some(openai("cedar", 2))); + assert_eq!(session.tts, test_snapshot().tts); + let saved = crate::saved_settings::load(&path).unwrap(); + assert_eq!(saved.tts, None); + assert_eq!( + saved.arguments, + [ + "--voice", + "Aaron", + "--language", + "en-US", + "--spokesperson-voice", + "cedar", + "--spokesperson-rate", + "1.25" + ] + .map(str::to_string) + ); + drop(actor); + child.wait().unwrap(); + } + #[test] fn pocket_tts_save_preserves_the_active_bundle_path() { let directory = tempfile::tempdir().unwrap(); diff --git a/src-tauri/crates/berd-call/src/main.rs b/src-tauri/crates/berd-call/src/main.rs index 217e8565c..5e9c639b5 100644 --- a/src-tauri/crates/berd-call/src/main.rs +++ b/src-tauri/crates/berd-call/src/main.rs @@ -36,8 +36,8 @@ use berd_call::openai_spokesperson::{ }; use berd_call::protocol::{ CancelOutcome, DismissHandoffsOutcome, ExpertTurnOutcome, InputDuringTtsOutcome, - NotAdmittedReason, OutputReadyOutcome, SessionMessage, SessionRequest, - SpokespersonVoiceSnapshot, TtsSettingsOutcome, VoiceSessionSnapshot, + NotAdmittedReason, OutputReadyOutcome, SessionMessage, SessionRequest, TtsSettingsOutcome, + VoiceSessionSnapshot, }; use berd_call::realtime_audio_delivery::RealtimeAudioDelivery; use berd_call::realtime_host_lifecycle::RealtimeSessionLossAction; @@ -192,6 +192,9 @@ struct SessionConfig { /// In Expert-Spokesperson mode, speak the Expert's replies with `tts` /// rather than having the Spokesperson say them. expert_tts: bool, + /// The Spokesperson's own voice and speed when replies have theirs. + spokesperson_voice: Option, + spokesperson_rate: Option, } #[derive(Clone, Debug, Default, PartialEq)] @@ -862,6 +865,14 @@ fn parse_saved_start_args_at(args: &[String], path: PathBuf) -> Result Result host_control::ControlRequest::Spokesperson { + voice: Some( + args.get(index + 1) + .filter(|value| !value.trim().is_empty()) + .ok_or("--spokesperson-voice requires a name")? + .clone(), + ), + rate: None, + }, + "--spokesperson-rate" => host_control::ControlRequest::Spokesperson { + voice: None, + rate: Some( + args.get(index + 1) + .and_then(|value| value.parse::().ok()) + .filter(|rate| rate.is_finite() && *rate > 0.0) + .ok_or("--spokesperson-rate requires a positive number")?, + ), + }, "--tts" => { let value = args .get(index + 1) @@ -1082,7 +1114,7 @@ fn parse_host_settings_args( } const SETTINGS_CHOICE_ERROR: &str = - "provide exactly one of --non-blocking, --rate, --voice, --tts, --input-during-tts, --muted, or --restart"; + "provide exactly one of --non-blocking, --rate, --voice, --spokesperson-voice, --spokesperson-rate, --tts, --input-during-tts, --muted, or --restart"; fn parse_bool_setting(flag: &str, value: Option<&String>) -> Result { match value.map(String::as_str) { @@ -2240,6 +2272,20 @@ fn run_session(config: SessionConfig, pcm_output_fd: RawFd) -> Result<(), String &mut writer, )?; } + Input::Request(SessionRequest::SetSpokespersonSettings { id, .. }) => { + write_message( + &mut writer, + &SessionMessage::TtsSettingsResult { + id, + outcome: TtsSettingsOutcome::Rejected, + snapshot: tts_slot + .as_ref() + .expect("hello initialized TTS") + .snapshot()?, + message: Some("a conventional call has no Spokesperson".into()), + }, + )?; + } Input::Request(SessionRequest::SetTtsSettings { id, expected_revision, @@ -3140,17 +3186,16 @@ fn apply_spokesperson_startup_settings( session: &SessionConfig, spokesperson: &mut OpenAiSpokespersonConfig, ) -> Result<(), String> { + // Replies with their own voice leave the Spokesperson's speed to it. let rate = match &session.tts { + _ if session.expert_tts => session.spokesperson_rate.unwrap_or(spokesperson.speed()), TtsBackendConfig::OpenAi { rate } | TtsBackendConfig::Siri { rate, .. } | TtsBackendConfig::Pocket { rate, .. } => *rate, }; - // A rate for the Expert's own voice need not suit the Spokesperson's. - let rate = if session.expert_tts { - rate.clamp(0.25, 1.5) - } else { - rate - }; + if let Some(voice) = &session.spokesperson_voice { + spokesperson.session.voice = Some(voice.clone()); + } if !(0.25..=1.5).contains(&rate) { return Err("Expert-Spokesperson rate must be between 0.25 and 1.5".into()); } @@ -4276,10 +4321,7 @@ fn run_expert_spokesperson_session( None => tts.clone(), }, input_during_tts: input_policy.snapshot()?, - spokesperson: Some(SpokespersonVoiceSnapshot { - model: spokesperson_config.model().into(), - voice: spokesperson_config.voice().into(), - }), + spokesperson: reply_tts.is_some().then(|| tts.clone()), }; let (created, events) = OpenAiSpokespersonRuntime::spawn_observed(spokesperson_config.clone())?; @@ -4814,11 +4856,18 @@ fn run_expert_spokesperson_session( }); } } - Input::Request(SessionRequest::SetTtsSettings { - id, - expected_revision, - settings, - }) => { + Input::Request( + SessionRequest::SetTtsSettings { + id, + expected_revision, + settings, + } + | SessionRequest::SetSpokespersonSettings { + id, + expected_revision, + settings, + }, + ) => { let request = PendingSpokespersonSettingsUpdate { id, base_revision: expected_revision, @@ -5739,6 +5788,8 @@ fn parse_args(args: &[String]) -> Result { let mut mode = SessionMode::Conventional; let mut endpoints = SessionEndpointOverrides::default(); let mut expert_tts = false; + let mut spokesperson_voice = None; + let mut spokesperson_rate = None; let mut index = 2; while index < args.len() { let flag = args[index].as_str(); @@ -5781,6 +5832,14 @@ fn parse_args(args: &[String]) -> Result { _ => return Err("--expert-voice must be tts or spokesperson".into()), } } + "--spokesperson-voice" => spokesperson_voice = Some(value.clone()), + "--spokesperson-rate" => { + spokesperson_rate = Some( + value + .parse::() + .map_err(|_| "--spokesperson-rate must be a number".to_string())?, + ) + } "--pcm-output-fd" => {} _ => return Err(format!("unknown argument: {flag}").into()), } @@ -5794,6 +5853,20 @@ fn parse_args(args: &[String]) -> Result { if expert_tts && mode != SessionMode::ExpertSpokesperson { return Err("--expert-voice requires --mode expert-spokesperson".into()); } + if (spokesperson_voice.is_some() || spokesperson_rate.is_some()) && !expert_tts { + return Err( + "--spokesperson-voice and --spokesperson-rate require --expert-voice tts".into(), + ); + } + if let Some(voice) = &spokesperson_voice { + if !berd_call::openai_realtime_protocol::OPENAI_REALTIME_VOICE_IDS.contains(&voice.as_str()) + { + return Err(format!("OpenAI has no realtime voice named {voice}").into()); + } + } + if spokesperson_rate.is_some_and(|rate| !(0.25..=1.5).contains(&rate)) { + return Err("--spokesperson-rate must be between 0.25 and 1.5".into()); + } if mode == SessionMode::ExpertSpokesperson && (endpoints.stt.is_some() || endpoints.tts.is_some() && !expert_tts) { @@ -5811,6 +5884,8 @@ fn parse_args(args: &[String]) -> Result { mode, endpoints, expert_tts, + spokesperson_voice, + spokesperson_rate, }) } @@ -7700,6 +7775,7 @@ fn validate_request(request: SessionRequest) -> Result { | SessionRequest::SetInputMuted { id, .. } | SessionRequest::SetConversationStatus { id, .. } | SessionRequest::SetTtsSettings { id, .. } + | SessionRequest::SetSpokespersonSettings { id, .. } | SessionRequest::SetInputDuringTts { id, .. } | SessionRequest::ResetInput { id } | SessionRequest::PrepareSpeak { id, .. } @@ -7770,6 +7846,10 @@ fn validate_request(request: SessionRequest) -> Result { SessionRequest::SetTtsSettings { expected_revision: 0, .. + } + | SessionRequest::SetSpokespersonSettings { + expected_revision: 0, + .. } => return Err("expected TTS revision must be positive".into()), SessionRequest::SetInputDuringTts { expected_revision: 0, @@ -10047,6 +10127,8 @@ mod tests { mode: SessionMode::Conventional, endpoints: SessionEndpointOverrides::default(), expert_tts: false, + spokesperson_voice: None, + spokesperson_rate: None, } ); @@ -10058,6 +10140,8 @@ mod tests { mode: SessionMode::Conventional, endpoints: SessionEndpointOverrides::default(), expert_tts: false, + spokesperson_voice: None, + spokesperson_rate: None, } ); } @@ -10135,6 +10219,8 @@ mod tests { mode: SessionMode::Conventional, endpoints: SessionEndpointOverrides::default(), expert_tts: false, + spokesperson_voice: None, + spokesperson_rate: None, } ); assert!(parse_args(&args(&[ @@ -10191,6 +10277,8 @@ mod tests { mode: SessionMode::ExpertSpokesperson, endpoints: SessionEndpointOverrides::default(), expert_tts: false, + spokesperson_voice: None, + spokesperson_rate: None, }; let mut realtime = OpenAiSpokespersonConfig { endpoint: "ws://localhost".into(), @@ -10256,7 +10344,35 @@ mod tests { semantic_transcript: Vec::new(), }; apply_spokesperson_startup_settings(&own, &mut realtime).unwrap(); - assert_eq!(realtime.speed(), 1.5); + // The replies' rate is theirs, not the Spokesperson's. + assert_eq!(realtime.speed(), 1.0); + let chosen = expert(&[ + "--tts-backend", + "openai", + "--expert-voice", + "tts", + "--spokesperson-voice", + "cedar", + "--spokesperson-rate", + "1.25", + ]) + .unwrap(); + apply_spokesperson_startup_settings(&chosen, &mut realtime).unwrap(); + assert_eq!((realtime.voice(), realtime.speed()), ("cedar", 1.25)); + for (extra, error) in [ + (&["--spokesperson-voice", "cedar"][..], "--expert-voice tts"), + ( + &["--expert-voice", "tts", "--spokesperson-voice", "nobody"], + "no realtime voice named nobody", + ), + ( + &["--expert-voice", "tts", "--spokesperson-rate", "2"], + "between 0.25 and 1.5", + ), + ] { + let all = [&["--tts-backend", "openai"][..], extra].concat(); + assert!(expert(&all).unwrap_err().contains(error), "{extra:?}"); + } let read_aloud = expert(&["--tts-backend", "openai", "--rate", "2.0"]).unwrap(); assert!(apply_spokesperson_startup_settings(&read_aloud, &mut realtime).is_err()); assert!(expert(&["--expert-voice", "loud"]) @@ -10385,6 +10501,8 @@ mod tests { mode: SessionMode::Conventional, endpoints: SessionEndpointOverrides::default(), expert_tts: false, + spokesperson_voice: None, + spokesperson_rate: None, } ); assert!(parse_args(&args(&[ @@ -10447,6 +10565,8 @@ mod tests { mode: SessionMode::Conventional, endpoints: SessionEndpointOverrides::default(), expert_tts: false, + spokesperson_voice: None, + spokesperson_rate: None, } ); assert!(parse_args(&args(&[ @@ -10892,6 +11012,8 @@ mod tests { mode: SessionMode::Conventional, endpoints: SessionEndpointOverrides::default(), expert_tts: false, + spokesperson_voice: None, + spokesperson_rate: None, } ); } @@ -11949,6 +12071,23 @@ mod tests { assert!(start(&["--no-menu-bar", "--no-menu-bar"]).is_err()); } + #[test] + fn spokesperson_settings_name_its_voice_or_rate() { + let parse = |flag: &str, value: &str| { + parse_host_settings_args(&args(&["berd-call", "settings", flag, value])) + .map(|(_, request)| request) + }; + assert!(matches!( + parse("--spokesperson-voice", "cedar").unwrap(), + host_control::ControlRequest::Spokesperson { voice: Some(voice), rate: None } if voice == "cedar" + )); + assert!(matches!( + parse("--spokesperson-rate", "1.25").unwrap(), + host_control::ControlRequest::Spokesperson { voice: None, rate: Some(rate) } if rate == 1.25 + )); + assert!(parse("--spokesperson-rate", "fast").is_err()); + } + #[test] fn settings_rate_changes_only_the_speech_rate() { let (_, request) = @@ -11959,6 +12098,51 @@ mod tests { ); } + #[test] + fn saved_spokesperson_voice_waits_for_calls_whose_replies_have_their_own() { + let directory = tempfile::tempdir().unwrap(); + let path = directory.path().join("settings.json"); + let chosen = args(&[ + "--spokesperson-voice", + "cedar", + "--spokesperson-rate", + "1.25", + ]); + saved_settings::save( + &path, + &saved_settings::SavedSettings { + arguments: [ + args(&["--voice", "Aaron", "--language", "en-US"]), + chosen.clone(), + ] + .concat(), + ..saved_settings::SavedSettings::default() + }, + ) + .unwrap(); + let start = |extra: &[&str]| { + parse_saved_start_args_at( + &args(&[&["berd-call", "start"], extra].concat()), + path.clone(), + ) + .unwrap() + }; + let conventional = start(&[]); + assert!(!conventional + .session_arguments + .iter() + .any(|flag| flag.starts_with("--spokesperson"))); + assert!(conventional.saved.unwrap().1.arguments.starts_with(&chosen)); + let own = start(&["--mode", "expert-spokesperson", "--expert-voice", "tts"]); + assert!(own + .session_arguments + .ends_with(&args(&["--expert-voice", "tts"]))); + assert!(own + .session_arguments + .windows(2) + .any(|pair| pair == ["--spokesperson-voice", "cedar"])); + } + #[test] fn saved_start_restores_preferences_but_not_agent_routing() { let directory = tempfile::tempdir().unwrap(); diff --git a/src-tauri/crates/berd-call/src/menu_bar.rs b/src-tauri/crates/berd-call/src/menu_bar.rs index 276370887..be5eebab1 100644 --- a/src-tauri/crates/berd-call/src/menu_bar.rs +++ b/src-tauri/crates/berd-call/src/menu_bar.rs @@ -28,6 +28,7 @@ use crate::host_control::{self, ControlRequest}; use crate::StartOptions; const RATES: [f32; 7] = [0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0]; +const SPOKESPERSON_RATES: std::ops::RangeInclusive = 0.25..=1.5; const REFRESH_INTERVAL: Duration = Duration::from_millis(100); /// Runs the call on a worker thread while the main thread owns the menu bar. @@ -64,6 +65,8 @@ pub(crate) enum MenuAction { language: Option, }, Rate(f32), + SpokespersonVoice(String), + SpokespersonRate(f32), Muted(bool), InputDuringTts(InputDuringTtsPolicy), Stop, @@ -121,33 +124,8 @@ fn menu_model(state: &MenuState) -> Vec { }; let muted = status["muted"].as_bool().unwrap_or(false); let suppressing = input_policy(status) == Some(InputDuringTtsPolicy::SuppressInput); - let rate = status["session"]["tts"]["rate"].as_f64().unwrap_or(1.0) as f32; - let rates = RATES - .iter() - .filter(|&&option| rate_range(status).contains(&option)) - .map(|&option| MenuNode { - checked: (option - rate).abs() < 0.01, - ..item(rate_title(option), Some(MenuAction::Rate(option))) - }) - .collect(); - let voice = status["session"]["tts"]["voice"] - .as_str() - .unwrap_or("Unknown"); - let language = status["session"]["tts"]["language"].as_str(); - let voices = state - .voices - .iter() - .map(|entry| MenuNode { - checked: entry.voice == voice && entry.language.as_deref() == language, - ..item( - entry.title.clone(), - Some(MenuAction::Voice { - voice: entry.voice.clone(), - language: entry.language.clone(), - }), - ) - }) - .collect(); + let tts = &status["session"]["tts"]; + let spokesperson = &status["session"]["spokesperson"]; let mut nodes = vec![item(format!("Berd Call · {}", mode_name(status)), None)]; for error in [&state.action_error, &state.catalog_error] .into_iter() @@ -155,16 +133,31 @@ fn menu_model(state: &MenuState) -> Vec { { nodes.push(item(format!("Control error: {error}"), None)); } - nodes.extend([ - SEPARATOR, - MenuNode { - children: voices, - ..item(format!("Voice: {voice}"), None) - }, - MenuNode { - children: rates, - ..item(format!("Speech Rate: {}", rate_title(rate)), None) + nodes.push(SEPARATOR); + // Replies with their own voice put the Spokesperson's beside it. + let agents = if spokesperson.is_null() { "" } else { "Agent " }; + nodes.extend(voice_menu( + agents, + tts, + &state.voices, + rate_range(status), + |entry| MenuAction::Voice { + voice: entry.voice.clone(), + language: entry.language.clone(), }, + MenuAction::Rate, + )); + if !spokesperson.is_null() { + nodes.extend(voice_menu( + "Spokesperson ", + spokesperson, + &available_voices("openai").unwrap_or_default(), + SPOKESPERSON_RATES, + |entry| MenuAction::SpokespersonVoice(entry.voice.clone()), + MenuAction::SpokespersonRate, + )); + } + nodes.extend([ MenuNode { checked: suppressing, ..item( @@ -188,6 +181,43 @@ fn menu_model(state: &MenuState) -> Vec { nodes } +/// A voice's choices and speeds, titled with whose voice it is. +fn voice_menu( + whose: &str, + tts: &Value, + voices: &[VoiceChoice], + range: std::ops::RangeInclusive, + choose: impl Fn(&VoiceChoice) -> MenuAction, + rate_action: impl Fn(f32) -> MenuAction, +) -> [MenuNode; 2] { + let voice = tts["voice"].as_str().unwrap_or("Unknown"); + let language = tts["language"].as_str(); + let rate = tts["rate"].as_f64().unwrap_or(1.0) as f32; + [ + MenuNode { + children: voices + .iter() + .map(|entry| MenuNode { + checked: entry.voice == voice && entry.language.as_deref() == language, + ..item(entry.title.clone(), Some(choose(entry))) + }) + .collect(), + ..item(format!("{whose}Voice: {voice}"), None) + }, + MenuNode { + children: RATES + .iter() + .filter(|option| range.contains(option)) + .map(|&option| MenuNode { + checked: (option - rate).abs() < 0.01, + ..item(rate_title(option), Some(rate_action(option))) + }) + .collect(), + ..item(format!("{whose}Speech Rate: {}", rate_title(rate)), None) + }, + ] +} + /// Translates a menu action into the control request the CLI would send. pub(crate) fn control_request(action: &MenuAction) -> ControlRequest { match action { @@ -196,6 +226,14 @@ pub(crate) fn control_request(action: &MenuAction) -> ControlRequest { language: language.clone(), }, MenuAction::Rate(rate) => ControlRequest::Rate { rate: *rate }, + MenuAction::SpokespersonVoice(voice) => ControlRequest::Spokesperson { + voice: Some(voice.clone()), + rate: None, + }, + MenuAction::SpokespersonRate(rate) => ControlRequest::Spokesperson { + voice: None, + rate: Some(*rate), + }, MenuAction::Muted(muted) => ControlRequest::Muted { muted: *muted }, MenuAction::InputDuringTts(policy) => ControlRequest::InputDuringTts { policy: *policy }, MenuAction::Stop => ControlRequest::Stop, @@ -204,8 +242,8 @@ pub(crate) fn control_request(action: &MenuAction) -> ControlRequest { /// The rates `berd-call` accepts for the call's TTS backend and mode. fn rate_range(status: &Value) -> std::ops::RangeInclusive { - if mode_name(status) == "Expert-Spokesperson" { - return 0.25..=1.5; + if mode_name(status) == "Expert-Spokesperson" && status["session"]["spokesperson"].is_null() { + return SPOKESPERSON_RATES; } match status["session"]["tts"]["backend"].as_str() { Some("siri") => 0.5..=2.0, @@ -546,7 +584,8 @@ fn action_domain(request: &ControlRequest) -> ActionDomain { match request { ControlRequest::TtsSettings { .. } | ControlRequest::Rate { .. } - | ControlRequest::Voice { .. } => ActionDomain::Tts, + | ControlRequest::Voice { .. } + | ControlRequest::Spokesperson { .. } => ActionDomain::Tts, ControlRequest::Muted { .. } => ActionDomain::Mute, ControlRequest::InputDuringTts { .. } => ActionDomain::InputDuringTts, ControlRequest::Stop => ActionDomain::Stop, @@ -560,6 +599,7 @@ fn revisioned_tts_action(request: &ControlRequest) -> bool { ControlRequest::TtsSettings { .. } | ControlRequest::Rate { .. } | ControlRequest::Voice { .. } + | ControlRequest::Spokesperson { .. } ) } @@ -737,6 +777,41 @@ mod tests { find(&menu_model(&MenuState::default()), "End Call"); } + #[test] + fn replies_with_their_own_voice_put_the_spokespersons_beside_it() { + let mut status = status(false, "allow_barge_in", Some("expert-spokesperson")); + status["session"]["spokesperson"] = json!({"backend": "openai", "model": "gpt-realtime", "voice": "marin", "rate": 1.0, "revision": 1}); + let nodes = menu_model(&MenuState { + status: Some(status), + ..MenuState::default() + }); + let titles = |wanted: &str| -> Vec { + find(&nodes, wanted) + .children + .iter() + .map(|node| node.title.clone()) + .collect() + }; + assert_eq!( + titles("Agent Speech Rate: 1.25×"), + ["0.5×", "0.75×", "1×", "1.25×", "1.5×", "1.75×", "2×"] + ); + assert_eq!( + titles("Spokesperson Speech Rate: 1×"), + ["0.5×", "0.75×", "1×", "1.25×", "1.5×"] + ); + let cedar = find(&nodes, "Spokesperson Voice: marin") + .children + .iter() + .find(|node| node.title == "cedar") + .unwrap(); + assert_eq!( + serde_json::to_value(control_request(cedar.action.as_ref().unwrap())).unwrap(), + json!({"command": "spokesperson", "voice": "cedar", "rate": null}) + ); + find(&nodes, "Agent Voice: Aaron"); + } + #[test] fn actions_become_the_same_requests_as_the_cli() { assert_eq!( diff --git a/src-tauri/crates/berd-call/src/protocol.rs b/src-tauri/crates/berd-call/src/protocol.rs index e27904bd0..550aaeaaa 100644 --- a/src-tauri/crates/berd-call/src/protocol.rs +++ b/src-tauri/crates/berd-call/src/protocol.rs @@ -32,6 +32,12 @@ pub enum SessionRequest { expected_revision: u64, settings: TtsSettings, }, + /// Changes the Spokesperson's voice in a session whose replies have their own. + SetSpokespersonSettings { + id: u64, + expected_revision: u64, + settings: TtsSettings, + }, SetInputDuringTts { id: u64, expected_revision: u64, @@ -161,15 +167,9 @@ pub struct VoiceSessionSnapshot { /// Spokesperson's own voice when it reads them. pub tts: TtsConfigurationSnapshot, pub input_during_tts: InputDuringTtsSnapshot, - /// The Spokesperson's voice, in Expert-Spokesperson sessions. + /// The Spokesperson's voice, when replies have their own. #[serde(default, skip_serializing_if = "Option::is_none")] - pub spokesperson: Option, -} - -#[derive(Clone, Debug, Deserialize, Serialize, PartialEq, Eq)] -pub struct SpokespersonVoiceSnapshot { - pub model: String, - pub voice: String, + pub spokesperson: Option, } #[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq, Eq)] @@ -413,14 +413,18 @@ mod tests { revision: 1, policy: InputDuringTtsPolicy::AllowBargeIn, }, - spokesperson: Some(SpokespersonVoiceSnapshot { - model: "gpt-realtime".into(), - voice: "cedar".into(), + spokesperson: Some(TtsConfigurationSnapshot { + revision: 1, + settings: TtsSettings::OpenAi { + model: "gpt-realtime".into(), + voice: "cedar".into(), + rate: 1.0, + }, }), }, }) .unwrap(), - r#"{"type":"ready","id":4,"protocol":5,"session":{"tts":{"revision":1,"backend":"openai","model":"gpt-4o-mini-tts","voice":"marin","rate":1.0},"input_during_tts":{"revision":1,"policy":"allow_barge_in"},"spokesperson":{"model":"gpt-realtime","voice":"cedar"}}}"# + r#"{"type":"ready","id":4,"protocol":5,"session":{"tts":{"revision":1,"backend":"openai","model":"gpt-4o-mini-tts","voice":"marin","rate":1.0},"input_during_tts":{"revision":1,"policy":"allow_barge_in"},"spokesperson":{"revision":1,"backend":"openai","model":"gpt-realtime","voice":"cedar","rate":1.0}}}"# ); assert_eq!( serde_json::from_str::( diff --git a/src-tauri/crates/berd-call/src/saved_settings.rs b/src-tauri/crates/berd-call/src/saved_settings.rs index 0caf56779..e9ff2f081 100644 --- a/src-tauri/crates/berd-call/src/saved_settings.rs +++ b/src-tauri/crates/berd-call/src/saved_settings.rs @@ -99,6 +99,8 @@ fn validate_arguments(arguments: &[String]) -> Result<(), String> { | "--rate" | "--model-dir" | "--stt-model-dir" + | "--spokesperson-voice" + | "--spokesperson-rate" ) }) { @@ -139,6 +141,32 @@ pub(crate) fn persistable_arguments(arguments: &[String]) -> Vec { .collect() } +fn spokesperson_flag(pair: &[String]) -> bool { + matches!( + pair[0].as_str(), + "--spokesperson-voice" | "--spokesperson-rate" + ) +} + +/// Leaves out the Spokesperson's own voice, which only calls whose replies have theirs take. +pub(crate) fn without_spokesperson(arguments: &[String]) -> Vec { + arguments + .chunks_exact(2) + .filter(|pair| !spokesperson_flag(pair)) + .flatten() + .cloned() + .collect() +} + +pub(crate) fn only_spokesperson(arguments: &[String]) -> Vec { + arguments + .chunks_exact(2) + .filter(|pair| spokesperson_flag(pair)) + .flatten() + .cloned() + .collect() +} + pub(crate) fn without_tts(arguments: &[String]) -> Vec { arguments .chunks_exact(2) diff --git a/src-tauri/crates/berd-call/tests/session_protocol.rs b/src-tauri/crates/berd-call/tests/session_protocol.rs index cd9ea0ca6..c86cc1147 100644 --- a/src-tauri/crates/berd-call/tests/session_protocol.rs +++ b/src-tauri/crates/berd-call/tests/session_protocol.rs @@ -2719,6 +2719,73 @@ fn spawn_speech_server() -> (String, mpsc::Receiver) { (url, receiver) } +#[test] +fn expert_voice_keeps_a_separate_spokesperson_voice_control() { + let (speech_url, _speech_requests) = spawn_speech_server(); + let (endpoint_tx, endpoint_rx) = mpsc::sync_channel(1); + let server = std::thread::spawn(move || { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap() + .block_on(async move { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + endpoint_tx + .send(format!("ws://{}/", listener.local_addr().unwrap())) + .unwrap(); + let (old_stream, _) = listener.accept().await.unwrap(); + let mut old = accept_async(old_stream).await.unwrap(); + let initial = receive_realtime_json(&mut old).await; + assert_eq!(initial["session"]["audio"]["output"]["voice"], "cedar"); + assert_eq!(initial["session"]["audio"]["output"]["speed"], 1.25); + acknowledge_realtime_session(&mut old, &initial).await; + + let (candidate_stream, _) = listener.accept().await.unwrap(); + let mut candidate = accept_async(candidate_stream).await.unwrap(); + let replacement = receive_realtime_json(&mut candidate).await; + assert_eq!( + replacement["session"]["audio"]["output"]["voice"], + "new-voice" + ); + assert_eq!(replacement["session"]["audio"]["output"]["speed"], 1.25); + acknowledge_realtime_session(&mut candidate, &replacement).await; + while receive_realtime_json(&mut old).await["type"] != "input_audio_buffer.clear" {} + send_realtime_json(&mut old, json!({"type":"input_audio_buffer.cleared"})).await; + while candidate.next().await.is_some() {} + }); + }); + let endpoint = endpoint_rx.recv().unwrap(); + let mut session = ExpertSpokespersonTestSession::start_with_args( + endpoint, + &[ + "--expert-voice", + "tts", + "--tts-url", + &speech_url, + "--spokesperson-voice", + "cedar", + "--spokesperson-rate", + "1.25", + ], + ); + assert_eq!(session.ready["session"]["tts"]["backend"], "openai"); + assert_eq!(session.ready["session"]["spokesperson"]["voice"], "cedar"); + session.send(json!({ + "type":"set_spokesperson_settings", + "id":2, + "expected_revision":1, + "settings":{"backend":"openai","model":"test-model","voice":"new-voice","rate":1.25} + })); + let applied = session.recv(Duration::from_secs(3)); + assert_eq!(applied["type"], "tts_settings_result", "{applied}"); + assert_eq!(applied["id"], 2); + assert_eq!(applied["outcome"], "applied", "{applied}"); + assert_eq!(applied["snapshot"]["revision"], 2); + assert_eq!(applied["snapshot"]["voice"], "new-voice"); + session.shutdown(); + server.join().unwrap(); +} + #[test] fn expert_voice_speaks_replies_itself_and_tells_the_spokesperson_what_was_heard() { let (speech_url, speech_requests) = spawn_speech_server(); @@ -2782,7 +2849,7 @@ fn expert_voice_speaks_replies_itself_and_tells_the_spokesperson_what_was_heard( assert_eq!(session.ready["session"]["tts"]["backend"], "openai"); assert_eq!( session.ready["session"]["spokesperson"], - json!({"model":"test-model","voice":"old-voice"}) + json!({"revision":1,"backend":"openai","model":"test-model","voice":"old-voice","rate":1.0}) ); assert_eq!(session.recv(Duration::from_secs(2))["origin"], "handoff"); assert_eq!( From e553b5f618f8b9456dbf16c0c88c64af39efa3c6 Mon Sep 17 00:00:00 2001 From: John Tennant Date: Mon, 19 Oct 2026 17:12:33 -0400 Subject: [PATCH 03/10] berd-call: let one reply speak in another voice `speak --voice NAME [--language BCP47]` speaks a single utterance in another voice of the call's reply backend, keeping its other settings and leaving the call's voice and saved settings alone. Agents sharing a call can then sound distinct. A voice the backend doesn't offer, or one for a call whose replies the Spokesperson speaks, is not admitted with the new `invalid_voice` reason. Built backends are cached per voice until the call's settings change. Co-Authored-By: Claude Opus 5.5 Signed-off-by: John Tennant --- src-tauri/crates/berd-call/PROTOCOL.md | 8 +- src-tauri/crates/berd-call/README.md | 4 + src-tauri/crates/berd-call/src/cli_help.rs | 6 +- .../crates/berd-call/src/configured_tts.rs | 95 ++++++++++ .../crates/berd-call/src/host_control.rs | 46 ++++- .../crates/berd-call/src/host_session.rs | 15 ++ src-tauri/crates/berd-call/src/main.rs | 165 ++++++++++++++++-- src-tauri/crates/berd-call/src/protocol.rs | 7 + .../berd-call/tests/session_protocol.rs | 16 +- 9 files changed, 340 insertions(+), 22 deletions(-) diff --git a/src-tauri/crates/berd-call/PROTOCOL.md b/src-tauri/crates/berd-call/PROTOCOL.md index 0d1d20639..410801034 100644 --- a/src-tauri/crates/berd-call/PROTOCOL.md +++ b/src-tauri/crates/berd-call/PROTOCOL.md @@ -206,7 +206,7 @@ it never admits a replacement while old host audio may still be active. {"type":"set_spokesperson_settings","id":u64,"expected_revision":u64,"settings":TtsSettings} {"type":"set_input_during_tts","id":u64,"expected_revision":u64,"policy":"allow_barge_in"|"suppress_input"} {"type":"reset_input","id":u64} -{"type":"prepare_speak","id":u64,"acknowledgement":u64|null,"text":string} +{"type":"prepare_speak","id":u64,"acknowledgement":u64|null,"text":string,"resolved_handoff_ids":[string]?,"voice":string?,"language":string?} {"type":"output_ready","id":u64,"speech_id":u64} audio acknowledgements listed above {"type":"query_state","id":u64,"after":u64} @@ -216,6 +216,10 @@ audio acknowledgements listed above ``` Unknown fields are rejected. IDs are positive. Speak text is at most 16 KiB. +`prepare_speak.voice` speaks that one utterance in another voice of the reply +backend, keeping its other settings; `language` changes a Siri voice's locale +too. A voice the backend doesn't offer, or a voice for a call whose replies +are spoken by the Spokesperson, is `not_admitted(invalid_voice)`. The parent cannot author speaking state or finalized input; those are derived only from PCM by the child runtime. @@ -348,7 +352,7 @@ cancellation and shutdown terminals. ```text {"type":"pending","id":u64,"utterances":[{"token":u64,"text":string}]} -{"type":"not_admitted","id":u64,"reason":"paused"|"in_progress"|"cancelled"|"empty_text"} +{"type":"not_admitted","id":u64,"reason":"paused"|"in_progress"|"cancelled"|"empty_text"|"invalid_handoff"|"invalid_voice"} {"type":"state","id":u64,"confirmed_token":u64,"utterances_after":[{"token":u64,"text":string}]} {"type":"cancel_result","id":u64,"outcome":"cancelled"|"stale","speech_id":u64|null} {"type":"speech_started","id":u64,"speech_id":u64} diff --git a/src-tauri/crates/berd-call/README.md b/src-tauri/crates/berd-call/README.md index d48af1818..049ca53e7 100644 --- a/src-tauri/crates/berd-call/README.md +++ b/src-tauri/crates/berd-call/README.md @@ -57,9 +57,13 @@ Start a call with the same backend flags accepted by `session`: berd-call start --voice Aaron --language en-US --stream berd-call status berd-call speak --re 7 "I found the cause." +berd-call speak --voice Nora "Sol here: the tests pass." berd-call stop ``` +`speak --voice` speaks one reply in another voice of the call's reply backend +without changing the call's voice, so agents sharing a call can sound distinct. + `start` stays in the foreground and owns the child session process. The control server listens only on `127.0.0.1:5222` by default; all four commands accept `--port` when more than one local call must be addressed. `--stream` writes one diff --git a/src-tauri/crates/berd-call/src/cli_help.rs b/src-tauri/crates/berd-call/src/cli_help.rs index 4d5edab41..64125af97 100644 --- a/src-tauri/crates/berd-call/src/cli_help.rs +++ b/src-tauri/crates/berd-call/src/cli_help.rs @@ -58,12 +58,14 @@ const SPEAK_HELP: &str = r#"Speak through the active local voice call. Usage: berd-call speak [--port PORT] [--re CURSOR] - [--resolves HANDOFF_ID]... TEXT + [--resolves HANDOFF_ID]... [--voice NAME [--language BCP47]] TEXT Speech waits for admission. The session setting controls whether it also waits for delivery. Non-blocking sessions require --stream or --codex and emit speech_result rows only for interruptions, failures, or pending user input. Interrupted results include -estimatedSpokenText, a best-effort prefix based on delivered audio."#; +estimatedSpokenText, a best-effort prefix based on delivered audio. --voice speaks +this text in another voice of the call's reply backend without changing its saved +voice, so several agents can share one call in distinct voices."#; const SETTINGS_HELP: &str = r#"Update settings for the active local voice call. diff --git a/src-tauri/crates/berd-call/src/configured_tts.rs b/src-tauri/crates/berd-call/src/configured_tts.rs index eaab4d423..c55f467f5 100644 --- a/src-tauri/crates/berd-call/src/configured_tts.rs +++ b/src-tauri/crates/berd-call/src/configured_tts.rs @@ -1,3 +1,4 @@ +use std::collections::HashMap; use std::fmt; use std::path::PathBuf; use std::sync::{Arc, Mutex}; @@ -214,6 +215,33 @@ impl TtsConfiguration { } } + /// This configuration in another of its backend's voices. + fn with_voice(mut self, name: &str, locale: Option<&str>) -> Result { + match &mut self { + Self::Siri { + voice, language, .. + } => { + *voice = name.into(); + if let Some(locale) = locale { + *language = locale.into(); + } + } + Self::OpenAi { voice, .. } | Self::Pocket { voice, .. } => { + if locale.is_some() { + return Err("only Siri voices take a language".into()); + } + *voice = name.into(); + } + } + if let Self::OpenAi { voice, .. } = &self { + if !crate::openai_realtime_protocol::OPENAI_REALTIME_VOICE_IDS.contains(&voice.as_str()) + { + return Err(format!("OpenAI has no voice named {voice}")); + } + } + Ok(self) + } + fn build(&self) -> Result, String> { validate_settings(&self.settings())?; match self { @@ -291,6 +319,8 @@ struct ConfiguredTtsState { configuration: TtsConfiguration, backend: Arc, snapshot: TtsConfigurationSnapshot, + /// Backends for other voices, built for single utterances at this revision. + voices: HashMap<(String, Option), TtsConfigurationLease>, } pub struct ConfiguredTtsSlot { @@ -364,6 +394,7 @@ impl ConfiguredTtsSlot { revision: 1, settings, }, + voices: HashMap::new(), }), } } @@ -386,6 +417,37 @@ impl ConfiguredTtsSlot { }) } + /// The configured backend speaking one utterance in another of its voices, + /// keeping the other settings. Building it doesn't hold the slot lock. + pub fn lease_voice( + &self, + voice: &str, + language: Option<&str>, + ) -> Result { + let key = (voice.to_string(), language.map(str::to_string)); + let poisoned = || "configured TTS lock was poisoned".to_string(); + let (configuration, revision) = { + let state = self.inner.lock().map_err(|_| poisoned())?; + if let Some(lease) = state.voices.get(&key) { + return Ok(lease.clone()); + } + (state.configuration.clone(), state.snapshot.revision) + }; + let configuration = configuration.with_voice(voice, language)?; + let lease = TtsConfigurationLease { + backend: configuration.build()?, + snapshot: TtsConfigurationSnapshot { + revision, + settings: configuration.settings(), + }, + }; + let mut state = self.inner.lock().map_err(|_| poisoned())?; + if state.snapshot.revision == revision { + state.voices.insert(key, lease.clone()); + } + Ok(lease) + } + /// Builds and validates a replacement without holding the slot lock. pub fn prepare_replacement( &self, @@ -475,6 +537,7 @@ impl ConfiguredTtsSlot { })?; state.configuration = replacement.configuration; state.backend = replacement.backend; + state.voices.clear(); state.snapshot = TtsConfigurationSnapshot { revision, settings: replacement.settings, @@ -554,6 +617,38 @@ mod tests { } } + #[test] + fn one_utterance_can_use_another_voice_without_changing_the_slot() { + let settings = |voice: &str| TtsSettings::OpenAi { + model: "gpt-4o-mini-tts".into(), + voice: voice.into(), + rate: 1.25, + }; + let slot = ConfiguredTtsSlot::new_for_test(settings("marin")); + let cedar = slot.lease_voice("cedar", None).unwrap(); + assert_eq!(cedar.snapshot().settings, settings("cedar")); + assert!(Arc::ptr_eq( + cedar.backend(), + slot.lease_voice("cedar", None).unwrap().backend() + )); + assert_eq!(slot.snapshot().unwrap().settings, settings("marin")); + assert!(slot + .lease_voice("nobody", None) + .err() + .unwrap() + .contains("no voice named nobody")); + assert!(slot + .lease_voice("cedar", Some("en-US")) + .err() + .unwrap() + .contains("language")); + slot.replace_for_test(1, settings("ash")).unwrap(); + assert!(!Arc::ptr_eq( + cedar.backend(), + slot.lease_voice("cedar", None).unwrap().backend() + )); + } + #[test] fn snapshot_is_sanitized_and_revisioned() { let slot = ConfiguredTtsSlot::new(TtsConfiguration::openai( diff --git a/src-tauri/crates/berd-call/src/host_control.rs b/src-tauri/crates/berd-call/src/host_control.rs index cd00eb516..e98cadecf 100644 --- a/src-tauri/crates/berd-call/src/host_control.rs +++ b/src-tauri/crates/berd-call/src/host_control.rs @@ -28,6 +28,8 @@ pub(crate) trait HostControl: Send + Sync + 'static { text: String, acknowledgement: Option, resolved_handoff_ids: Vec, + voice: Option, + language: Option, ) -> Result; fn stop(&self) -> Result; fn set_non_blocking(&self, enabled: bool) -> Result; @@ -115,6 +117,10 @@ pub(crate) enum ControlRequest { acknowledgement: Option, #[serde(rename = "resolvedHandoffIds")] resolved_handoff_ids: Vec, + #[serde(default, skip_serializing_if = "Option::is_none")] + voice: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + language: Option, }, Stop, NonBlocking { @@ -231,9 +237,16 @@ fn handle_connection(mut stream: TcpStream, control: &dyn HostControl) -> Result text, acknowledgement, resolved_handoff_ids, + voice, + language, } => { - validate_speak(&text, &resolved_handoff_ids)?; - control.speak(text, acknowledgement, resolved_handoff_ids) + validate_speak( + &text, + &resolved_handoff_ids, + voice.as_deref(), + language.is_some(), + )?; + control.speak(text, acknowledgement, resolved_handoff_ids, voice, language) } ControlRequest::Stop => control.stop(), ControlRequest::TtsSettings { settings } => control.set_tts(settings), @@ -310,7 +323,15 @@ fn write_json_line( .map_err(|error| format!("could not write berd-call {label}: {error}")) } -fn validate_speak(text: &str, resolved_handoff_ids: &[String]) -> Result<(), String> { +fn validate_speak( + text: &str, + resolved_handoff_ids: &[String], + voice: Option<&str>, + language: bool, +) -> Result<(), String> { + if voice.is_some_and(|voice| voice.trim().is_empty()) || language && voice.is_none() { + return Err("speak --language requires a nonempty --voice".into()); + } if text.trim().is_empty() { return Err("speak text must not be empty".into()); } @@ -390,11 +411,15 @@ mod tests { text: String, acknowledgement: Option, resolved_handoff_ids: Vec, + voice: Option, + language: Option, ) -> Result { Ok(json!({ "spoken":text, "acknowledgement":acknowledgement, "resolvedHandoffIds":resolved_handoff_ids, + "voice":voice, + "language":language, })) } @@ -441,6 +466,8 @@ mod tests { text: "hello".into(), acknowledgement: Some(7), resolved_handoff_ids: vec!["call-1".into()], + voice: Some("Nora".into()), + language: Some("en-GB".into()), } ) .unwrap(), @@ -448,6 +475,8 @@ mod tests { "spoken":"hello", "acknowledgement":7, "resolvedHandoffIds":["call-1"], + "voice":"Nora", + "language":"en-GB", }) ); assert_eq!( @@ -459,9 +488,12 @@ mod tests { #[test] fn speak_validation_rejects_invalid_values() { - assert!(validate_speak(" ", &[]).is_err()); - assert!(validate_speak(&"x".repeat(16 * 1024 + 1), &[]).is_err()); - assert!(validate_speak("hello", &["".into()]).is_err()); - assert!(validate_speak("hello", &["same".into(), "same".into()]).is_err()); + assert!(validate_speak(" ", &[], None, false).is_err()); + assert!(validate_speak(&"x".repeat(16 * 1024 + 1), &[], None, false).is_err()); + assert!(validate_speak("hello", &["".into()], None, false).is_err()); + assert!(validate_speak("hello", &["same".into(), "same".into()], None, false).is_err()); + assert!(validate_speak("hello", &[], None, true).is_err()); + assert!(validate_speak("hello", &[], Some(" "), false).is_err()); + assert!(validate_speak("hello", &[], Some("Nora"), true).is_ok()); } } diff --git a/src-tauri/crates/berd-call/src/host_session.rs b/src-tauri/crates/berd-call/src/host_session.rs index ac01d0fa5..71b09af77 100644 --- a/src-tauri/crates/berd-call/src/host_session.rs +++ b/src-tauri/crates/berd-call/src/host_session.rs @@ -517,6 +517,8 @@ enum ControlCommand { text: String, acknowledgement: Option, resolved_handoff_ids: Vec, + voice: Option, + language: Option, non_blocking: bool, response: SyncSender>, }, @@ -739,6 +741,8 @@ impl HostControl for SessionControl { text: String, acknowledgement: Option, resolved_handoff_ids: Vec, + voice: Option, + language: Option, ) -> Result { let (tx, rx) = mpsc::sync_channel(1); self.commands @@ -746,6 +750,8 @@ impl HostControl for SessionControl { text, acknowledgement, resolved_handoff_ids, + voice, + language, non_blocking: self.non_blocking.load(Ordering::SeqCst), response: tx, }) @@ -1092,6 +1098,8 @@ impl SessionActor { text, acknowledgement, resolved_handoff_ids, + voice, + language, non_blocking, response, } => { @@ -1104,6 +1112,8 @@ impl SessionActor { text, acknowledgement, resolved_handoff_ids, + voice, + language, non_blocking, response, }); @@ -1117,6 +1127,8 @@ impl SessionActor { acknowledgement, text: text.clone(), resolved_handoff_ids, + voice, + language, }, )?; self.pending_speak = Some(PendingSpeak { @@ -2661,6 +2673,7 @@ fn not_admitted_reason_name(reason: NotAdmittedReason) -> &'static str { NotAdmittedReason::Cancelled => "cancelled", NotAdmittedReason::EmptyText => "empty_text", NotAdmittedReason::InvalidHandoff => "invalid_handoff", + NotAdmittedReason::InvalidVoice => "invalid_voice", } } @@ -3356,6 +3369,8 @@ mod tests { text: "next".into(), acknowledgement: Some(7), resolved_handoff_ids: Vec::new(), + voice: None, + language: None, non_blocking: true, response: next_tx, }) diff --git a/src-tauri/crates/berd-call/src/main.rs b/src-tauri/crates/berd-call/src/main.rs index 5e9c639b5..ebac91061 100644 --- a/src-tauri/crates/berd-call/src/main.rs +++ b/src-tauri/crates/berd-call/src/main.rs @@ -539,6 +539,8 @@ fn main() { port, acknowledgement, resolved_handoff_ids, + voice, + language, text, } = parse_or_exit(parse_speak_control_args(&args), &args); let response = host_control::request( @@ -547,6 +549,8 @@ fn main() { text, acknowledgement, resolved_handoff_ids, + voice, + language, }, ) .unwrap_or_else(|error| operational_error("speak", error)); @@ -921,6 +925,8 @@ struct SpeakOptions { port: u16, acknowledgement: Option, resolved_handoff_ids: Vec, + voice: Option, + language: Option, text: String, } @@ -929,6 +935,7 @@ fn parse_speak_control_args(args: &[String]) -> Result Result return Err("--re may be provided only once".into()), + flag @ ("--voice" | "--language") => { + let slot = if flag == "--voice" { + &mut voice + } else { + &mut language + }; + if slot.is_some() { + return Err(format!("{flag} may be provided only once").into()); + } + *slot = Some( + args.get(index + 1) + .filter(|value| !value.trim().is_empty()) + .ok_or_else(|| format!("{flag} requires a value"))? + .clone(), + ); + index += 2; + } "--resolves" => { let value = args .get(index + 1) @@ -984,10 +1008,15 @@ fn parse_speak_control_args(args: &[String]) -> Result MAX_SPEAK_TEXT_BYTES { return Err("speak text is larger than 16 KiB".into()); } + if language.is_some() && voice.is_none() { + return Err("--language requires --voice".into()); + } Ok(SpeakOptions { port, acknowledgement, resolved_handoff_ids, + voice, + language, text, }) } @@ -1992,7 +2021,7 @@ fn run_session(config: SessionConfig, pcm_output_fd: RawFd) -> Result<(), String let mut next_input_token = 1_u64; let mut pending_control = None; let mut processed_pcm = 0_u64; - let mut held: Option = None; + let mut held: Option = None; let mut active: Option = None; let mut status_sound_runtime = StatusSoundRuntime::default(); @@ -2132,7 +2161,7 @@ fn run_session(config: SessionConfig, pcm_output_fd: RawFd) -> Result<(), String "session is shutting down", &mut writer, )?; - if let Some(held) = held.take() { + if let Some((held, _)) = held.take() { write_message( &mut writer, &SessionMessage::NotAdmitted { @@ -2366,12 +2395,19 @@ fn run_session(config: SessionConfig, pcm_output_fd: RawFd) -> Result<(), String acknowledgement, text, resolved_handoff_ids: _, + voice, + language, }) => { let request = PrepareRequest { id, acknowledgement, text, }; + let Some(voice) = + speak_voice(tts_slot.as_deref(), voice, language, id, &mut writer)? + else { + continue; + }; if held.is_some() { write_message( &mut writer, @@ -2383,6 +2419,7 @@ fn run_session(config: SessionConfig, pcm_output_fd: RawFd) -> Result<(), String } else { process_prepare( request, + voice, &mut core, tts_slot.as_deref().expect("hello initialized TTS"), input_during_tts_slot @@ -2681,6 +2718,8 @@ struct PendingExpertPrepare { acknowledgement: Option, text: String, resolved_handoff_ids: Vec, + /// The Expert's voice for this reply, when not its usual one. + voice: Option, } struct PendingSpokespersonSettingsUpdate { @@ -2919,6 +2958,8 @@ struct LiveResponse { /// An Expert reply spoken in its own voice, with the handoffs it answers. /// It has no provider response; its whole text is played at once. reply: Option>, + /// The voice it asked for, when not the Expert's usual one. + voice: Option, delivery: RealtimeAudioDelivery, pending_audio: VecDeque>, pending_frames: usize, @@ -2937,6 +2978,7 @@ impl LiveResponse { speech_id, expert_text: None, reply: None, + voice: None, delivery: RealtimeAudioDelivery::default(), pending_audio: VecDeque::new(), pending_frames: 0, @@ -4373,12 +4415,20 @@ fn run_expert_spokesperson_session( acknowledgement, text, resolved_handoff_ids, + voice, + language, }) => { + let Some(voice) = + speak_voice(reply_tts.as_deref(), voice, language, id, &mut writer)? + else { + continue; + }; let request = PendingExpertPrepare { id, acknowledgement, text, resolved_handoff_ids, + voice, }; let routing = route_expert_prepare( &mut turn_gate, @@ -5153,6 +5203,7 @@ fn submit_expert_prepare( let mut response = LiveResponse::new(Some(request.id), Some(speech_id)); response.expert_text = Some(message); response.reply = Some(resolved_handoff_ids); + response.voice = request.voice; response.audio_done = true; response.finish_sent = true; response.server_finished = true; @@ -5334,7 +5385,13 @@ fn start_next_live_playback( response.speech_id.expect("an Expert reply has a speech ID"), response.prepare_id, response.expert_text.clone().unwrap_or_default(), - Arc::clone(tts.lease()?.backend()), + Arc::clone( + response + .voice + .clone() + .map_or_else(|| tts.lease(), Ok)? + .backend(), + ), Arc::clone(audio_transport), audio_control_tx.clone(), reply_tx.clone(), @@ -7079,20 +7136,55 @@ fn reject_tts_configuration_update( ) } +/// A held reply, with the voice it asked for. +type HeldPrepare = (PrepareRequest, Option); + +/// The voice one reply asked for, or `None` after telling the host it can't +/// have it. +fn speak_voice( + slot: Option<&ConfiguredTtsSlot>, + voice: Option, + language: Option, + id: u64, + writer: &mut impl Write, +) -> Result>, String> { + let Some(voice) = voice else { + return Ok(Some(None)); + }; + let lease = slot + .ok_or_else(|| "this call's replies have no voice of their own".to_string()) + .and_then(|slot| slot.lease_voice(&voice, language.as_deref())); + match lease { + Ok(lease) => Ok(Some(Some(lease))), + Err(message) => { + eprintln!("berd-call: speech {id} can't use voice {voice}: {message}"); + write_message( + writer, + &SessionMessage::NotAdmitted { + id, + reason: NotAdmittedReason::InvalidVoice, + }, + )?; + Ok(None) + } + } +} + #[allow(clippy::too_many_arguments)] fn process_prepare( request: PrepareRequest, + voice: Option, core: &mut SessionCore, tts_slot: &ConfiguredTtsSlot, input_during_tts_slot: &InputDuringTtsSlot, active: &mut Option, - held: &mut Option, + held: &mut Option, writer: &mut impl Write, ) -> Result<(), String> { let id = request.id; match core.prepare(request.clone()) { PrepareOutcome::Hold => { - *held = Some(request); + *held = Some((request, voice)); } PrepareOutcome::Pending(utterances) => { write_message(writer, &SessionMessage::Pending { id, utterances })?; @@ -7105,7 +7197,7 @@ fn process_prepare( confirmed_token, text, } => { - let tts = tts_slot.lease()?; + let tts = voice.map_or_else(|| tts_slot.lease(), Ok)?; let input_during_tts = input_during_tts_slot.snapshot()?; *active = Some(ActivePlayback { prepare_id: id, @@ -7133,7 +7225,7 @@ fn process_prepare( } fn reevaluate_held( - held: &mut Option, + held: &mut Option, core: &mut SessionCore, tts_slot: Option<&ConfiguredTtsSlot>, input_during_tts_slot: Option<&InputDuringTtsSlot>, @@ -7141,9 +7233,10 @@ fn reevaluate_held( writer: &mut impl Write, ) -> Result<(), String> { if !core.user_speaking() && !core.recognition_pending() && active.is_none() { - if let Some(pending_prepare) = held.take() { + if let Some((pending_prepare, voice)) = held.take() { process_prepare( pending_prepare, + voice, core, tts_slot.expect("held prepare requires initialized TTS"), input_during_tts_slot @@ -7478,12 +7571,12 @@ fn handle_reset_input( fn handle_cancel( id: u64, - held: &mut Option, + held: &mut Option, core: &mut SessionCore, active: &mut Option, writer: &mut impl Write, ) -> Result<(), String> { - if held.as_ref().is_some_and(|held| held.id == id) { + if held.as_ref().is_some_and(|(held, _)| held.id == id) { held.take(); write_message( writer, @@ -8257,6 +8350,7 @@ mod tests { acknowledgement: Some(1), text: "answer".into(), resolved_handoff_ids: Vec::new(), + voice: None, }; assert!(matches!( gate.defer_if_busy(request, false, 1), @@ -8296,6 +8390,7 @@ mod tests { acknowledgement: Some(1), text: "first".into(), resolved_handoff_ids: Vec::new(), + voice: None, }, false, 1, @@ -8309,6 +8404,7 @@ mod tests { acknowledgement: Some(1), text: "second".into(), resolved_handoff_ids: Vec::new(), + voice: None, }, false, 1, @@ -8339,6 +8435,7 @@ mod tests { acknowledgement: Some(1), text: "second".into(), resolved_handoff_ids: Vec::new(), + voice: None, }, expert_output_reserved(&directives, &HashSet::new(), None, &responses), false, @@ -8865,6 +8962,7 @@ mod tests { acknowledgement: Some(1), text: "answer after the user".into(), resolved_handoff_ids: Vec::new(), + voice: None, }, false, 1, @@ -9289,6 +9387,35 @@ mod tests { test_tts_slot().lease().unwrap() } + #[test] + fn speech_in_another_voice_is_refused_when_the_voice_is_unavailable() { + let slot = test_tts_slot(); + let mut writer = Vec::new(); + assert!(speak_voice(Some(&slot), None, None, 1, &mut writer) + .unwrap() + .unwrap() + .is_none()); + let coral = speak_voice(Some(&slot), Some("coral".into()), None, 2, &mut writer) + .unwrap() + .unwrap() + .unwrap(); + assert_eq!(coral.snapshot().settings.voice(), "coral"); + assert!(writer.is_empty()); + for (id, slot, voice) in [(3, Some(&slot), "nobody"), (4, None, "coral")] { + assert!(speak_voice(slot, Some(voice.into()), None, id, &mut writer) + .unwrap() + .is_none()); + } + let refusals = String::from_utf8(writer).unwrap(); + assert_eq!( + refusals.lines().collect::>(), + [ + r#"{"type":"not_admitted","id":3,"reason":"invalid_voice"}"#, + r#"{"type":"not_admitted","id":4,"reason":"invalid_voice"}"#, + ] + ); + } + fn test_input_policy_slot() -> InputDuringTtsSlot { InputDuringTtsSlot::new(InputDuringTtsPolicy::AllowBargeIn) } @@ -11194,6 +11321,8 @@ mod tests { acknowledgement: None, text: "answer".into(), resolved_handoff_ids: vec![" ".into()], + voice: None, + language: None, }; assert_eq!( validate_request(empty).unwrap_err(), @@ -11244,6 +11373,7 @@ mod tests { acknowledgement: None, text: "reply".into(), }, + None, &mut core, &test_tts_slot(), &input_policy, @@ -11289,6 +11419,7 @@ mod tests { acknowledgement: None, text: "old voice".into(), }, + None, &mut core, &slot, &input_policy, @@ -12377,6 +12508,10 @@ mod tests { "5300", "--re", "17", + "--voice", + "Nora", + "--language", + "en-GB", "hello", ])) .unwrap(), @@ -12384,9 +12519,19 @@ mod tests { port: 5300, acknowledgement: Some(17), resolved_handoff_ids: Vec::new(), + voice: Some("Nora".into()), + language: Some("en-GB".into()), text: "hello".into(), } ); + assert!(parse_speak_control_args(&args(&[ + "berd-call", + "speak", + "--language", + "en-GB", + "hello" + ])) + .is_err()); assert_eq!( parse_control_port(&args(&["berd-call", "status", "--port", "5301"])).unwrap(), 5301 diff --git a/src-tauri/crates/berd-call/src/protocol.rs b/src-tauri/crates/berd-call/src/protocol.rs index 550aaeaaa..02e58c261 100644 --- a/src-tauri/crates/berd-call/src/protocol.rs +++ b/src-tauri/crates/berd-call/src/protocol.rs @@ -52,6 +52,10 @@ pub enum SessionRequest { text: String, #[serde(default)] resolved_handoff_ids: Vec, + #[serde(default, skip_serializing_if = "Option::is_none")] + voice: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + language: Option, }, OutputReady { id: u64, @@ -145,6 +149,7 @@ pub enum NotAdmittedReason { Cancelled, EmptyText, InvalidHandoff, + InvalidVoice, } #[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq, Eq)] @@ -394,6 +399,8 @@ mod tests { acknowledgement: Some(0), text: "hi".into(), resolved_handoff_ids: Vec::new(), + voice: None, + language: None, } ); assert_eq!( diff --git a/src-tauri/crates/berd-call/tests/session_protocol.rs b/src-tauri/crates/berd-call/tests/session_protocol.rs index c86cc1147..d2af1fa6c 100644 --- a/src-tauri/crates/berd-call/tests/session_protocol.rs +++ b/src-tauri/crates/berd-call/tests/session_protocol.rs @@ -2862,7 +2862,8 @@ fn expert_voice_speaks_replies_itself_and_tells_the_spokesperson_what_was_heard( "id":2, "acknowledgement":1, "text":"I checked the repository.", - "resolved_handoff_ids":["handoff-external-1"] + "resolved_handoff_ids":["handoff-external-1"], + "voice":"coral" })); let admitted = session.recv(Duration::from_secs(2)); assert_eq!(admitted["type"], "admitted", "{admitted}"); @@ -2889,6 +2890,19 @@ fn expert_voice_speaks_replies_itself_and_tells_the_spokesperson_what_was_heard( .recv_timeout(Duration::from_secs(1)) .unwrap(); assert_eq!(speech["input"], "I checked the repository."); + assert_eq!(speech["voice"], "coral"); + session.send(json!({ + "type":"prepare_speak", + "id":3, + "acknowledgement":1, + "text":"Who am I?", + "voice":"nobody" + })); + let rejected = session.recv(Duration::from_secs(2)); + assert_eq!( + rejected, + json!({"type":"not_admitted","id":3,"reason":"invalid_voice"}) + ); let (context, mentions, after) = context_rx.recv_timeout(Duration::from_secs(2)).unwrap(); assert_eq!(context["type"], "conversation.item.create", "{context}"); From 54f6b853a391c3a30d383fbcfd27475c5b99d23d Mon Sep 17 00:00:00 2001 From: John Tennant Date: Mon, 19 Oct 2026 17:17:56 -0400 Subject: [PATCH 04/10] berd-call: list Siri voices whose assets are missing as not installed sirittsd reports en-GB-C and en-GB-D as downloaded on a Mac where their assets aren't on disk, and then speaks both in the same low-quality substitute (byte-identical renders). Every voice that really speaks has a TTSAXResourceManager contentPath; these have none. The catalog now counts a voice as installed only when sirittsd reports it downloaded and its content path exists, so the menu bar, Berd's settings and the plugin stop offering it, and `voices download` fetches it instead. Co-Authored-By: Claude Opus 5.5 Signed-off-by: John Tennant --- src-tauri/crates/berd-call/README.md | 4 +++- src-tauri/crates/berd-call/native/siri_tts_bridge.m | 8 +++++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/src-tauri/crates/berd-call/README.md b/src-tauri/crates/berd-call/README.md index 049ca53e7..77cb30a97 100644 --- a/src-tauri/crates/berd-call/README.md +++ b/src-tauri/crates/berd-call/README.md @@ -270,7 +270,9 @@ retries cleanup. This avoids reporting a failed install after the model has already been applied. `voices.list` returns `backend`, normalized `languageFilter`, -`availableLanguages`, and exact voice records. `voices.download` returns the +`availableLanguages`, and exact voice records. A voice is `installed` only when +sirittsd reports it downloaded and its assets are on disk: sirittsd can claim +a voice whose assets are missing, then silently speak it in another voice. `voices.download` returns the canonical voice, `installed: true`, and `availabilityWaitSeconds`; a missing exact catalog identity fails with `voice_not_found` before any native download request. macOS status and install results contain `supported`, `locale`, diff --git a/src-tauri/crates/berd-call/native/siri_tts_bridge.m b/src-tauri/crates/berd-call/native/siri_tts_bridge.m index ce06b7a13..1f73dc5da 100644 --- a/src-tauri/crates/berd-call/native/siri_tts_bridge.m +++ b/src-tauri/crates/berd-call/native/siri_tts_bridge.m @@ -997,10 +997,14 @@ static void BerdDownloadedVoices( NSString *name = [resource valueForKey:@"name"] ?: @""; if (!name.length || !language.length) continue; NSString *key = [NSString stringWithFormat:@"%@|%@", name, normalizedLanguage]; + // sirittsd can report a voice as downloaded whose assets aren't on disk, + // and then speaks it in another voice. + NSString *content = [resource valueForKey:@"contentPath"]; byKey[key] = @{ @"name" : name, @"language" : language, @"sizeBytes" : [resource valueForKey:@"assetSize"] ?: @0, + @"content" : @(content.length && [NSFileManager.defaultManager fileExistsAtPath:content]), }; } return [[byKey allValues] sortedArrayUsingComparator: @@ -1142,7 +1146,9 @@ static BOOL BerdTriggerDownload(NSString *language, NSError **error) { NSString *key = [NSString stringWithFormat:@"%@|%@", candidate[@"name"], normalizedLanguage]; NSMutableDictionary *voice = [candidate mutableCopy]; - voice[@"installed"] = installedSnapshot[key] ?: @NO; + [voice removeObjectForKey:@"content"]; + voice[@"installed"] = + [installedSnapshot[key] boolValue] && [candidate[@"content"] boolValue] ? @YES : @NO; [result addObject:voice]; } NSData *json = [NSJSONSerialization dataWithJSONObject:result options:0 error:&error]; From d7568a6792edf68cd1a851d47e5eebdab693c7cf Mon Sep 17 00:00:00 2001 From: John Tennant Date: Tue, 20 Oct 2026 11:21:49 -0400 Subject: [PATCH 05/10] End the session when an Expert reply's audio never settles A reply whose playback failed without the host confirming the cancellation left old audio possibly still playing, yet the session went on admitting output. Carry the playback failure's quiescence into the reply result and end the session after reporting the failure, as conventional playback already does. Signed-off-by: John Tennant Co-Authored-By: Claude Opus 5.5 --- src-tauri/crates/berd-call/src/main.rs | 27 ++++--- .../berd-call/tests/session_protocol.rs | 77 +++++++++++++++++++ 2 files changed, 94 insertions(+), 10 deletions(-) diff --git a/src-tauri/crates/berd-call/src/main.rs b/src-tauri/crates/berd-call/src/main.rs index ebac91061..baf47a42c 100644 --- a/src-tauri/crates/berd-call/src/main.rs +++ b/src-tauri/crates/berd-call/src/main.rs @@ -2699,7 +2699,7 @@ type LivePlaybackResult = Result<(String, u64), (String, u64, String)>; enum ReplyOutcome { Completed, Interrupted { spoken_through_utf8: u64 }, - Failed(String), + Failed(PlaybackFailure), } type ReplyResult = (String, u64, ReplyOutcome); @@ -4141,6 +4141,7 @@ fn run_expert_spokesperson_session( .prepare_id .expect("an Expert reply has a prepare ID"); let text = response.expert_text.take().unwrap_or_default(); + let mut output_quiescent = true; let (message, heard) = match outcome { ReplyOutcome::Completed => ( SessionMessage::SpeechCompleted { id, speech_id }, @@ -4156,14 +4157,17 @@ fn run_expert_spokesperson_session( }, usize::try_from(spoken_through_utf8).unwrap_or(text.len()), ), - ReplyOutcome::Failed(message) => ( - SessionMessage::SpeechFailed { - id, - speech_id, - message, - }, - 0, - ), + ReplyOutcome::Failed(failure) => { + output_quiescent = failure.output_quiescent; + ( + SessionMessage::SpeechFailed { + id, + speech_id, + message: failure.message, + }, + 0, + ) + } }; let heard = (0..=heard.min(text.len())) .rev() @@ -4173,6 +4177,9 @@ fn run_expert_spokesperson_session( if response.claim_speech_terminal() { write_message(&mut writer, &message)?; } + if !output_quiescent { + return Err("remote PCM output did not reach a quiescent terminal".into()); + } finish_expert_reply( &mut core, runtime.as_ref().expect("initialized runtime"), @@ -4995,7 +5002,7 @@ fn spawn_reply_playback( spoken_through_utf8: u64::try_from(estimated_spoken_through_utf8(&text, &delivery)) .expect("speech text is bounded well below u64"), }, - Err(failure) => ReplyOutcome::Failed(failure.message), + Err(failure) => ReplyOutcome::Failed(failure), }; let _ = completed.send((worker_response_id, speech_id, outcome)); }); diff --git a/src-tauri/crates/berd-call/tests/session_protocol.rs b/src-tauri/crates/berd-call/tests/session_protocol.rs index d2af1fa6c..573e7720c 100644 --- a/src-tauri/crates/berd-call/tests/session_protocol.rs +++ b/src-tauri/crates/berd-call/tests/session_protocol.rs @@ -38,6 +38,8 @@ struct ExpertSpokespersonTestSession { struct AudioCancellationGate { observed: mpsc::SyncSender<()>, release: mpsc::Receiver<()>, + /// Hold the drain instead, as a hung host would, without reading anything after it. + at_drain: bool, } impl ExpertSpokespersonTestSession { @@ -391,6 +393,11 @@ fn spawn_audio_host_with_played_limit( let frames = u64::from_le_bytes(payload[16..24].try_into().unwrap()); assert_eq!(current, Some((speech_id, sequence, frames))); current = None; + if let Some(gate) = cancellation_gate.take_if(|gate| gate.at_drain) { + let _ = gate.observed.send(()); + let _ = gate.release.recv(); + return; + } json!({"type":"audio_drained","speech_id":speech_id,"sequence":sequence,"played_frames":frames}) } 4 => { @@ -949,6 +956,7 @@ fn autonomous_spokesperson_cancel_is_correlated_before_audio_cancellation_and_th Some(AudioCancellationGate { observed: cancel_observed_tx, release: release_cancel_rx, + at_drain: false, }), ); let started = loop { @@ -2786,6 +2794,75 @@ fn expert_voice_keeps_a_separate_spokesperson_voice_control() { server.join().unwrap(); } +#[test] +fn expert_voice_reply_whose_audio_never_settles_ends_the_session() { + let (speech_url, _speech_requests) = spawn_speech_server(); + let (endpoint_tx, endpoint_rx) = mpsc::sync_channel(1); + std::thread::spawn(move || { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap() + .block_on(async move { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + endpoint_tx + .send(format!("ws://{}/", listener.local_addr().unwrap())) + .unwrap(); + let (stream, _) = listener.accept().await.unwrap(); + let mut socket = accept_async(stream).await.unwrap(); + let initial = receive_realtime_json(&mut socket).await; + acknowledge_realtime_session(&mut socket, &initial).await; + while socket.next().await.is_some() {} + }); + }); + let (observed, cancel_observed) = mpsc::sync_channel(1); + let (release_cancel, release) = mpsc::sync_channel(1); + // The host never confirms the drain, nor the cancellation that follows it. + let mut session = ExpertSpokespersonTestSession::launch( + endpoint_rx.recv().unwrap(), + None, + None, + None, + Some(AudioCancellationGate { + observed, + release, + at_drain: true, + }), + &["--expert-voice", "tts", "--tts-url", &speech_url], + ); + session.send(json!({ + "type":"prepare_speak", + "id":2, + "acknowledgement":null, + "text":"I checked the repository." + })); + let admitted = session.recv(Duration::from_secs(2)); + assert_eq!(admitted["type"], "admitted", "{admitted}"); + session.send(json!({"type":"output_ready","id":2,"speech_id":admitted["speech_id"]})); + let deadline = std::time::Instant::now() + Duration::from_secs(8); + let mut messages = Vec::new(); + loop { + let remaining = deadline + .checked_duration_since(std::time::Instant::now()) + .unwrap_or_else(|| panic!("the session went on: {messages:?}")); + match session.output.recv_timeout(remaining) { + Ok(message) => messages.push(message), + Err(mpsc::RecvTimeoutError::Disconnected) => break, + Err(mpsc::RecvTimeoutError::Timeout) => panic!("the session went on: {messages:?}"), + } + } + assert!( + messages + .iter() + .any(|message| message["type"] == "speech_failed" && message["id"] == 2), + "the reply did not fail: {messages:?}" + ); + let status = session.child.0.as_mut().unwrap().wait().unwrap(); + assert!(!status.success()); + cancel_observed.try_recv().unwrap(); + release_cancel.send(()).unwrap(); +} + #[test] fn expert_voice_speaks_replies_itself_and_tells_the_spokesperson_what_was_heard() { let (speech_url, speech_requests) = spawn_speech_server(); From b1ced00bb45570800a04aadfe0159c01019775bb Mon Sep 17 00:00:00 2001 From: John Tennant Date: Tue, 20 Oct 2026 11:25:31 -0400 Subject: [PATCH 06/10] Build an utterance's new voice off the session loop `speak --voice` built and validated a voice it hadn't used yet on the session loop. Siri validation can wait seconds for sirittsd, so a reply already playing could miss its audio acknowledgement deadlines and microphone input could back up. Build the voice on a worker and hand the request back to the loop once it's ready, or refuse it there if the voice can't be built. Signed-off-by: John Tennant Co-Authored-By: Claude Opus 5.5 --- .../crates/berd-call/src/configured_tts.rs | 9 ++ src-tauri/crates/berd-call/src/main.rs | 116 ++++++++++++++++-- 2 files changed, 117 insertions(+), 8 deletions(-) diff --git a/src-tauri/crates/berd-call/src/configured_tts.rs b/src-tauri/crates/berd-call/src/configured_tts.rs index c55f467f5..09d156b96 100644 --- a/src-tauri/crates/berd-call/src/configured_tts.rs +++ b/src-tauri/crates/berd-call/src/configured_tts.rs @@ -417,6 +417,15 @@ impl ConfiguredTtsSlot { }) } + /// Whether `lease_voice` already has this voice built. + pub fn has_voice(&self, voice: &str, language: Option<&str>) -> bool { + self.inner.lock().is_ok_and(|state| { + state + .voices + .contains_key(&(voice.to_string(), language.map(str::to_string))) + }) + } + /// The configured backend speaking one utterance in another of its voices, /// keeping the other settings. Building it doesn't hold the slot lock. pub fn lease_voice( diff --git a/src-tauri/crates/berd-call/src/main.rs b/src-tauri/crates/berd-call/src/main.rs index baf47a42c..f7158c982 100644 --- a/src-tauri/crates/berd-call/src/main.rs +++ b/src-tauri/crates/berd-call/src/main.rs @@ -110,6 +110,12 @@ enum Input { Request(SessionRequest), Pcm(Box), Invalid(String), + /// A speech request's voice could not be built. + VoiceUnavailable { + id: u64, + voice: String, + message: String, + }, Eof, } @@ -2002,6 +2008,7 @@ fn expert_session_status_cues_suppressed( fn run_session(config: SessionConfig, pcm_output_fd: RawFd) -> Result<(), String> { let (control_tx, control_rx) = mpsc::channel(); let (pcm_tx, pcm_rx) = mpsc::sync_channel(INPUT_QUEUE_CAPACITY); + let voice_tx = control_tx.clone(); thread::spawn(move || read_framed_requests(io::stdin().lock(), control_tx, pcm_tx)); let audio_transport = Arc::new(unsafe { AudioPipeTransport::from_raw_fd(pcm_output_fd)? }); let (playback_tx, playback_rx) = mpsc::channel(); @@ -2107,6 +2114,9 @@ fn run_session(config: SessionConfig, pcm_output_fd: RawFd) -> Result<(), String ) else { continue; }; + let Some(input) = build_voice_later(input, tts_slot.as_ref(), &voice_tx) else { + continue; + }; match input { Input::Invalid(message) => { write_protocol_fatal(&mut writer, "invalid session input", &message)?; @@ -2116,6 +2126,9 @@ fn run_session(config: SessionConfig, pcm_output_fd: RawFd) -> Result<(), String } return Ok(()); } + Input::VoiceUnavailable { id, voice, message } => { + refuse_voice(id, &voice, &message, &mut writer)?; + } Input::Eof => { if let Some(current) = active.as_mut() { if let Some(flag) = ¤t.active { @@ -3251,6 +3264,7 @@ fn run_expert_spokesperson_session( ) -> Result<(), String> { let (control_tx, control_rx) = mpsc::channel(); let (pcm_tx, pcm_rx) = mpsc::sync_channel(INPUT_QUEUE_CAPACITY); + let voice_tx = control_tx.clone(); thread::spawn(move || read_framed_requests(io::stdin().lock(), control_tx, pcm_tx)); let audio_transport = Arc::new(unsafe { AudioPipeTransport::from_raw_fd(pcm_output_fd)? }); let (audio_control_tx, audio_control_rx) = mpsc::channel(); @@ -4244,11 +4258,17 @@ fn run_expert_spokesperson_session( ) else { continue; }; + let Some(input) = build_voice_later(input, reply_tts.as_ref(), &voice_tx) else { + continue; + }; match input { Input::Invalid(message) => { write_protocol_fatal(&mut writer, "invalid session input", &message)?; break; } + Input::VoiceUnavailable { id, voice, message } => { + refuse_voice(id, &voice, &message, &mut writer)?; + } Input::Eof => break, Input::Request(SessionRequest::Shutdown) => { if let Some(request) = queued_tts_settings.take() { @@ -7164,19 +7184,65 @@ fn speak_voice( match lease { Ok(lease) => Ok(Some(Some(lease))), Err(message) => { - eprintln!("berd-call: speech {id} can't use voice {voice}: {message}"); - write_message( - writer, - &SessionMessage::NotAdmitted { - id, - reason: NotAdmittedReason::InvalidVoice, - }, - )?; + refuse_voice(id, &voice, &message, writer)?; Ok(None) } } } +fn refuse_voice( + id: u64, + voice: &str, + message: &str, + writer: &mut impl Write, +) -> Result<(), String> { + eprintln!("berd-call: speech {id} can't use voice {voice}: {message}"); + write_message( + writer, + &SessionMessage::NotAdmitted { + id, + reason: NotAdmittedReason::InvalidVoice, + }, + ) +} + +/// Builds a speech request's new voice on a worker, since that can take seconds, so the session +/// keeps serving audio and input meanwhile; the worker hands the request back once it's built. +fn build_voice_later( + input: Input, + slot: Option<&Arc>, + controls: &mpsc::Sender, +) -> Option { + let ( + Some(slot), + Input::Request(SessionRequest::PrepareSpeak { + id, + voice: Some(voice), + language, + .. + }), + ) = (slot, &input) + else { + return Some(input); + }; + if slot.has_voice(voice, language.as_deref()) { + return Some(input); + } + let (id, voice, language) = (*id, voice.clone(), language.clone()); + let (slot, controls) = (Arc::clone(slot), controls.clone()); + thread::spawn(move || { + let input = match slot.lease_voice(&voice, language.as_deref()) { + Ok(_) => input, + Err(message) => Input::VoiceUnavailable { id, voice, message }, + }; + let _ = controls.send(OrderedControl { + after_pcm: 0, + input, + }); + }); + None +} + #[allow(clippy::too_many_arguments)] fn process_prepare( request: PrepareRequest, @@ -9394,6 +9460,40 @@ mod tests { test_tts_slot().lease().unwrap() } + #[test] + fn a_new_voice_is_built_off_the_session_loop_and_its_request_handed_back() { + let slot = Arc::new(test_tts_slot()); + let (controls, handed_back) = mpsc::channel::(); + let prepare = |id, voice: &str| { + Input::Request(SessionRequest::PrepareSpeak { + id, + acknowledgement: None, + text: "Hello.".into(), + resolved_handoff_ids: Vec::new(), + voice: Some(voice.into()), + language: None, + }) + }; + let next = || { + handed_back + .recv_timeout(Duration::from_secs(2)) + .unwrap() + .input + }; + assert!(build_voice_later(prepare(1, "coral"), Some(&slot), &controls).is_none()); + assert!(matches!( + next(), + Input::Request(SessionRequest::PrepareSpeak { id: 1, .. }) + )); + assert!(matches!( + build_voice_later(prepare(2, "coral"), Some(&slot), &controls), + Some(Input::Request(SessionRequest::PrepareSpeak { id: 2, .. })) + )); + assert!(build_voice_later(prepare(3, "nobody"), Some(&slot), &controls).is_none()); + assert!(matches!(next(), Input::VoiceUnavailable { id: 3, .. })); + assert!(build_voice_later(prepare(4, "coral"), None, &controls).is_some()); + } + #[test] fn speech_in_another_voice_is_refused_when_the_voice_is_unavailable() { let slot = test_tts_slot(); From 04b73fa88910461f492a4d57b7efeba6dd80b4aa Mon Sep 17 00:00:00 2001 From: John Tennant Date: Tue, 20 Oct 2026 11:33:02 -0400 Subject: [PATCH 07/10] Let cancel reach speech waiting on its voice A speech request whose voice was still being built was invisible to `cancel`, which answered stale; the request then came back and played. Track the requests waiting on a voice, cancel them there, and drop a cancelled one when its voice arrives. Signed-off-by: John Tennant Co-Authored-By: Claude Opus 5.5 --- src-tauri/crates/berd-call/src/main.rs | 194 +++++++++++++++++++------ 1 file changed, 146 insertions(+), 48 deletions(-) diff --git a/src-tauri/crates/berd-call/src/main.rs b/src-tauri/crates/berd-call/src/main.rs index f7158c982..3ee3b3138 100644 --- a/src-tauri/crates/berd-call/src/main.rs +++ b/src-tauri/crates/berd-call/src/main.rs @@ -110,6 +110,8 @@ enum Input { Request(SessionRequest), Pcm(Box), Invalid(String), + /// A speech request whose voice is now built. + VoiceBuilt(SessionRequest), /// A speech request's voice could not be built. VoiceUnavailable { id: u64, @@ -2008,7 +2010,10 @@ fn expert_session_status_cues_suppressed( fn run_session(config: SessionConfig, pcm_output_fd: RawFd) -> Result<(), String> { let (control_tx, control_rx) = mpsc::channel(); let (pcm_tx, pcm_rx) = mpsc::sync_channel(INPUT_QUEUE_CAPACITY); - let voice_tx = control_tx.clone(); + let mut voice_builds = VoiceBuilds { + controls: control_tx.clone(), + pending: HashSet::new(), + }; thread::spawn(move || read_framed_requests(io::stdin().lock(), control_tx, pcm_tx)); let audio_transport = Arc::new(unsafe { AudioPipeTransport::from_raw_fd(pcm_output_fd)? }); let (playback_tx, playback_rx) = mpsc::channel(); @@ -2114,7 +2119,7 @@ fn run_session(config: SessionConfig, pcm_output_fd: RawFd) -> Result<(), String ) else { continue; }; - let Some(input) = build_voice_later(input, tts_slot.as_ref(), &voice_tx) else { + let Some(input) = voice_builds.filter(input, tts_slot.as_ref(), &mut writer)? else { continue; }; match input { @@ -2126,8 +2131,8 @@ fn run_session(config: SessionConfig, pcm_output_fd: RawFd) -> Result<(), String } return Ok(()); } - Input::VoiceUnavailable { id, voice, message } => { - refuse_voice(id, &voice, &message, &mut writer)?; + Input::VoiceBuilt(_) | Input::VoiceUnavailable { .. } => { + unreachable!("voice builds are filtered") } Input::Eof => { if let Some(current) = active.as_mut() { @@ -3264,7 +3269,10 @@ fn run_expert_spokesperson_session( ) -> Result<(), String> { let (control_tx, control_rx) = mpsc::channel(); let (pcm_tx, pcm_rx) = mpsc::sync_channel(INPUT_QUEUE_CAPACITY); - let voice_tx = control_tx.clone(); + let mut voice_builds = VoiceBuilds { + controls: control_tx.clone(), + pending: HashSet::new(), + }; thread::spawn(move || read_framed_requests(io::stdin().lock(), control_tx, pcm_tx)); let audio_transport = Arc::new(unsafe { AudioPipeTransport::from_raw_fd(pcm_output_fd)? }); let (audio_control_tx, audio_control_rx) = mpsc::channel(); @@ -4258,7 +4266,7 @@ fn run_expert_spokesperson_session( ) else { continue; }; - let Some(input) = build_voice_later(input, reply_tts.as_ref(), &voice_tx) else { + let Some(input) = voice_builds.filter(input, reply_tts.as_ref(), &mut writer)? else { continue; }; match input { @@ -4266,8 +4274,8 @@ fn run_expert_spokesperson_session( write_protocol_fatal(&mut writer, "invalid session input", &message)?; break; } - Input::VoiceUnavailable { id, voice, message } => { - refuse_voice(id, &voice, &message, &mut writer)?; + Input::VoiceBuilt(_) | Input::VoiceUnavailable { .. } => { + unreachable!("voice builds are filtered") } Input::Eof => break, Input::Request(SessionRequest::Shutdown) => { @@ -7206,41 +7214,92 @@ fn refuse_voice( ) } -/// Builds a speech request's new voice on a worker, since that can take seconds, so the session -/// keeps serving audio and input meanwhile; the worker hands the request back once it's built. -fn build_voice_later( - input: Input, - slot: Option<&Arc>, - controls: &mpsc::Sender, -) -> Option { - let ( - Some(slot), - Input::Request(SessionRequest::PrepareSpeak { - id, - voice: Some(voice), - language, - .. - }), - ) = (slot, &input) +/// Speech requests waiting on a worker to build a voice the session hasn't used yet, since that +/// can take seconds and the session must keep serving audio and input meanwhile. +struct VoiceBuilds { + controls: mpsc::Sender, + pending: HashSet, +} + +impl VoiceBuilds { + /// Takes the requests that wait on a voice, and hands each back once its voice is built. + fn filter( + &mut self, + input: Input, + slot: Option<&Arc>, + writer: &mut impl Write, + ) -> Result, String> { + match input { + Input::VoiceBuilt(request) => { + let SessionRequest::PrepareSpeak { id, .. } = request else { + unreachable!("only speech waits on a voice") + }; + Ok(self.pending.remove(&id).then_some(Input::Request(request))) + } + Input::VoiceUnavailable { id, voice, message } => { + if self.pending.remove(&id) { + refuse_voice(id, &voice, &message, writer)?; + } + Ok(None) + } + Input::Request(SessionRequest::Cancel { id }) if self.pending.remove(&id) => { + write_message( + writer, + &SessionMessage::CancelResult { + id, + outcome: CancelOutcome::Cancelled, + speech_id: None, + }, + )?; + write_message( + writer, + &SessionMessage::NotAdmitted { + id, + reason: NotAdmittedReason::Cancelled, + }, + )?; + Ok(None) + } + Input::Request(request @ SessionRequest::PrepareSpeak { id, .. }) => { + let missing = slot.and_then(|slot| { + missing_voice(slot, &request).map(|voice| (Arc::clone(slot), voice)) + }); + let Some((slot, (voice, language))) = missing else { + return Ok(Some(Input::Request(request))); + }; + let controls = self.controls.clone(); + self.pending.insert(id); + thread::spawn(move || { + let input = match slot.lease_voice(&voice, language.as_deref()) { + Ok(_) => Input::VoiceBuilt(request), + Err(message) => Input::VoiceUnavailable { id, voice, message }, + }; + let _ = controls.send(OrderedControl { + after_pcm: 0, + input, + }); + }); + Ok(None) + } + input => Ok(Some(input)), + } + } +} + +/// The voice and language a speech request asks for, when the slot hasn't built them yet. +fn missing_voice( + slot: &ConfiguredTtsSlot, + request: &SessionRequest, +) -> Option<(String, Option)> { + let SessionRequest::PrepareSpeak { + voice: Some(voice), + language, + .. + } = request else { - return Some(input); + return None; }; - if slot.has_voice(voice, language.as_deref()) { - return Some(input); - } - let (id, voice, language) = (*id, voice.clone(), language.clone()); - let (slot, controls) = (Arc::clone(slot), controls.clone()); - thread::spawn(move || { - let input = match slot.lease_voice(&voice, language.as_deref()) { - Ok(_) => input, - Err(message) => Input::VoiceUnavailable { id, voice, message }, - }; - let _ = controls.send(OrderedControl { - after_pcm: 0, - input, - }); - }); - None + (!slot.has_voice(voice, language.as_deref())).then(|| (voice.clone(), language.clone())) } #[allow(clippy::too_many_arguments)] @@ -9464,6 +9523,12 @@ mod tests { fn a_new_voice_is_built_off_the_session_loop_and_its_request_handed_back() { let slot = Arc::new(test_tts_slot()); let (controls, handed_back) = mpsc::channel::(); + let mut builds = VoiceBuilds { + controls, + pending: HashSet::new(), + }; + let mut writer = Vec::new(); + let mut filter = |input, slot| builds.filter(input, slot, &mut writer).unwrap(); let prepare = |id, voice: &str| { Input::Request(SessionRequest::PrepareSpeak { id, @@ -9480,18 +9545,51 @@ mod tests { .unwrap() .input }; - assert!(build_voice_later(prepare(1, "coral"), Some(&slot), &controls).is_none()); + assert!(filter(prepare(1, "coral"), Some(&slot)).is_none()); assert!(matches!( - next(), - Input::Request(SessionRequest::PrepareSpeak { id: 1, .. }) + filter(next(), Some(&slot)), + Some(Input::Request(SessionRequest::PrepareSpeak { id: 1, .. })) )); assert!(matches!( - build_voice_later(prepare(2, "coral"), Some(&slot), &controls), + filter(prepare(2, "coral"), Some(&slot)), Some(Input::Request(SessionRequest::PrepareSpeak { id: 2, .. })) )); - assert!(build_voice_later(prepare(3, "nobody"), Some(&slot), &controls).is_none()); - assert!(matches!(next(), Input::VoiceUnavailable { id: 3, .. })); - assert!(build_voice_later(prepare(4, "coral"), None, &controls).is_some()); + assert!(filter(prepare(3, "nobody"), Some(&slot)).is_none()); + assert!(filter(next(), Some(&slot)).is_none()); + // A request cancelled while its voice is built is not revived by the build. + assert!(filter(prepare(4, "marin"), Some(&slot)).is_none()); + assert!(filter( + Input::Request(SessionRequest::Cancel { id: 4 }), + Some(&slot) + ) + .is_none()); + assert!(filter(next(), Some(&slot)).is_none()); + assert!(filter(prepare(5, "cedar"), None).is_some()); + drop(filter); + let types: Vec<_> = String::from_utf8(writer) + .unwrap() + .lines() + .map(|line| { + let message: serde_json::Value = serde_json::from_str(line).unwrap(); + format!( + "{} {} {}", + message["type"], + message["id"], + message["reason"] + .as_str() + .or(message["outcome"].as_str()) + .unwrap() + ) + }) + .collect(); + assert_eq!( + types, + [ + r#""not_admitted" 3 invalid_voice"#, + r#""cancel_result" 4 cancelled"#, + r#""not_admitted" 4 cancelled"#, + ] + ); } #[test] From a9696911fa00ce95fcece04657cbd287cab6ab32 Mon Sep 17 00:00:00 2001 From: John Tennant Date: Tue, 20 Oct 2026 11:34:20 -0400 Subject: [PATCH 08/10] Test that user speech and cancel stop an Expert reply The test host can now hold a drain and then keep serving, so a reply stays playing until something stops it. Signed-off-by: John Tennant Co-Authored-By: Claude Opus 5.5 --- .../berd-call/tests/session_protocol.rs | 105 +++++++++++++++++- 1 file changed, 100 insertions(+), 5 deletions(-) diff --git a/src-tauri/crates/berd-call/tests/session_protocol.rs b/src-tauri/crates/berd-call/tests/session_protocol.rs index 573e7720c..0228b9f67 100644 --- a/src-tauri/crates/berd-call/tests/session_protocol.rs +++ b/src-tauri/crates/berd-call/tests/session_protocol.rs @@ -38,7 +38,8 @@ struct ExpertSpokespersonTestSession { struct AudioCancellationGate { observed: mpsc::SyncSender<()>, release: mpsc::Receiver<()>, - /// Hold the drain instead, as a hung host would, without reading anything after it. + /// Hold the drain instead, reading nothing after it, as a hung host would. Once released, + /// the host goes on without ever draining that speech. at_drain: bool, } @@ -392,12 +393,14 @@ fn spawn_audio_host_with_played_limit( let sequence = u64::from_le_bytes(payload[8..16].try_into().unwrap()); let frames = u64::from_le_bytes(payload[16..24].try_into().unwrap()); assert_eq!(current, Some((speech_id, sequence, frames))); - current = None; if let Some(gate) = cancellation_gate.take_if(|gate| gate.at_drain) { let _ = gate.observed.send(()); - let _ = gate.release.recv(); - return; + if gate.release.recv().is_err() { + return; + } + continue; } + current = None; json!({"type":"audio_drained","speech_id":speech_id,"sequence":sequence,"played_frames":frames}) } 4 => { @@ -2794,6 +2797,98 @@ fn expert_voice_keeps_a_separate_spokesperson_voice_control() { server.join().unwrap(); } +enum ReplyInterruption { + UserSpeech, + Cancel, +} + +/// Stops an Expert reply partway through playback and returns what the session reports. +fn interrupt_expert_reply(interruption: ReplyInterruption) -> (Value, usize) { + let (speech_url, _speech_requests) = spawn_speech_server(); + let (endpoint_tx, endpoint_rx) = mpsc::sync_channel(1); + let (user_speaks, user_speech) = mpsc::sync_channel(1); + std::thread::spawn(move || { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap() + .block_on(async move { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + endpoint_tx + .send(format!("ws://{}/", listener.local_addr().unwrap())) + .unwrap(); + let (stream, _) = listener.accept().await.unwrap(); + let mut socket = accept_async(stream).await.unwrap(); + let initial = receive_realtime_json(&mut socket).await; + acknowledge_realtime_session(&mut socket, &initial).await; + if user_speech.recv().is_ok() { + send_realtime_json( + &mut socket, + json!({"type":"input_audio_buffer.speech_started","item_id":"user-1"}), + ) + .await; + } + while socket.next().await.is_some() {} + }); + }); + let (observed, playing) = mpsc::sync_channel(1); + let (release_drain, release) = mpsc::sync_channel(1); + // The host has played part of the reply and is still playing it. + let mut session = ExpertSpokespersonTestSession::launch( + endpoint_rx.recv().unwrap(), + None, + Some(1_000), + None, + Some(AudioCancellationGate { + observed, + release, + at_drain: true, + }), + &["--expert-voice", "tts", "--tts-url", &speech_url], + ); + let text = "I checked the repository and found three failing tests in the parser."; + session.send(json!({"type":"prepare_speak","id":2,"acknowledgement":null,"text":text})); + let admitted = session.recv(Duration::from_secs(2)); + assert_eq!(admitted["type"], "admitted", "{admitted}"); + session.send(json!({"type":"output_ready","id":2,"speech_id":admitted["speech_id"]})); + playing.recv_timeout(Duration::from_secs(2)).unwrap(); + release_drain.send(()).unwrap(); + match interruption { + ReplyInterruption::UserSpeech => user_speaks.send(()).unwrap(), + ReplyInterruption::Cancel => session.send(json!({"type":"cancel","id":2})), + } + let interrupted = loop { + let message = session.recv(Duration::from_secs(2)); + assert!( + !matches!( + message["type"].as_str(), + Some("fatal" | "speech_failed" | "speech_completed") + ), + "{message}" + ); + if message["type"] == "speech_interrupted" { + break message; + } + }; + drop(user_speaks); + session.shutdown(); + (interrupted, text.len()) +} + +#[test] +fn user_speech_interrupts_an_expert_reply_where_it_was_heard() { + let (interrupted, length) = interrupt_expert_reply(ReplyInterruption::UserSpeech); + assert_eq!(interrupted["id"], 2); + assert!(interrupted["spoken_through_utf8"].as_u64().unwrap() < length as u64); +} + +#[test] +fn cancel_interrupts_an_expert_reply_where_it_was_heard() { + let (interrupted, length) = interrupt_expert_reply(ReplyInterruption::Cancel); + assert_eq!(interrupted["id"], 2); + assert!(interrupted["spoken_through_utf8"].as_u64().unwrap() < length as u64); +} + #[test] fn expert_voice_reply_whose_audio_never_settles_ends_the_session() { let (speech_url, _speech_requests) = spawn_speech_server(); @@ -2860,7 +2955,7 @@ fn expert_voice_reply_whose_audio_never_settles_ends_the_session() { let status = session.child.0.as_mut().unwrap().wait().unwrap(); assert!(!status.success()); cancel_observed.try_recv().unwrap(); - release_cancel.send(()).unwrap(); + drop(release_cancel); } #[test] From 4d1cc303f3483846787d66ca59eefbddd9215c7d Mon Sep 17 00:00:00 2001 From: John Tennant Date: Tue, 20 Oct 2026 13:15:11 -0400 Subject: [PATCH 09/10] Wait for output_ready before playing an Expert reply An admitted Expert-voice reply was queued for playback at once, so it could start before the parent sent output_ready. It now waits: a matching output_ready marks it ready and queues it, and the accepted result is written before playback can begin. Mismatched, repeated or dropped requests are stale. Signed-off-by: John Tennant Co-Authored-By: Claude Opus 5.5 --- src-tauri/crates/berd-call/src/main.rs | 36 ++++++---- .../berd-call/tests/session_protocol.rs | 68 +++++++++++++++++++ 2 files changed, 92 insertions(+), 12 deletions(-) diff --git a/src-tauri/crates/berd-call/src/main.rs b/src-tauri/crates/berd-call/src/main.rs index 3ee3b3138..1cd81fce9 100644 --- a/src-tauri/crates/berd-call/src/main.rs +++ b/src-tauri/crates/berd-call/src/main.rs @@ -2978,6 +2978,8 @@ struct LiveResponse { reply: Option>, /// The voice it asked for, when not the Expert's usual one. voice: Option, + /// Whether the parent has accepted the reply's output, so it may play. + output_ready: bool, delivery: RealtimeAudioDelivery, pending_audio: VecDeque>, pending_frames: usize, @@ -2997,6 +2999,7 @@ impl LiveResponse { expert_text: None, reply: None, voice: None, + output_ready: false, delivery: RealtimeAudioDelivery::default(), pending_audio: VecDeque::new(), pending_frames: 0, @@ -4245,7 +4248,6 @@ fn run_expert_spokesperson_session( &mut directive_speeches, reply_tts.is_some(), &mut responses, - &mut waiting_responses, &mut next_speech_id, &mut writer, )?; @@ -4486,7 +4488,6 @@ fn run_expert_spokesperson_session( &mut directive_speeches, reply_tts.is_some(), &mut responses, - &mut waiting_responses, &mut next_speech_id, &mut writer, )?, @@ -4501,12 +4502,24 @@ fn run_expert_spokesperson_session( } } Input::Request(SessionRequest::OutputReady { id, speech_id }) => { - let valid = directive_speeches - .get(&id) - .is_some_and(|directive| directive.speech_id == speech_id) - || responses.values().any(|response| { - response.prepare_id == Some(id) && response.speech_id == Some(speech_id) - }); + let matches = |response: &LiveResponse| { + response.prepare_id == Some(id) && response.speech_id == Some(speech_id) + }; + let reply = responses.iter_mut().find(|(_, response)| { + response.reply.is_some() && !response.output_ready && matches(response) + }); + let valid = if let Some((response_id, response)) = reply { + response.output_ready = true; + waiting_responses.push_back(response_id.clone()); + true + } else { + directive_speeches + .get(&id) + .is_some_and(|directive| directive.speech_id == speech_id) + || responses + .values() + .any(|response| response.reply.is_none() && matches(response)) + }; write_message( &mut writer, &SessionMessage::OutputReadyResult { @@ -5191,7 +5204,6 @@ fn submit_expert_prepare( directive_speeches: &mut HashMap, expert_tts: bool, responses: &mut HashMap, - waiting_responses: &mut VecDeque, next_speech_id: &mut u64, writer: &mut impl Write, ) -> Result<(), String> { @@ -5233,7 +5245,8 @@ fn submit_expert_prepare( let resolved_handoff_ids = request.resolved_handoff_ids; core.mark_handoffs_resolving(&resolved_handoff_ids)?; if expert_tts { - // Played in turn with the Spokesperson's responses, without one of its own. + // Played in turn with the Spokesperson's responses, without one of its own, + // once the parent is ready for its output. let response_id = format!("expert-reply-{speech_id}"); let mut response = LiveResponse::new(Some(request.id), Some(speech_id)); response.expert_text = Some(message); @@ -5242,8 +5255,7 @@ fn submit_expert_prepare( response.audio_done = true; response.finish_sent = true; response.server_finished = true; - responses.insert(response_id.clone(), response); - waiting_responses.push_back(response_id); + responses.insert(response_id, response); } else { directive_speeches.insert( request.id, diff --git a/src-tauri/crates/berd-call/tests/session_protocol.rs b/src-tauri/crates/berd-call/tests/session_protocol.rs index 0228b9f67..090b3d2c2 100644 --- a/src-tauri/crates/berd-call/tests/session_protocol.rs +++ b/src-tauri/crates/berd-call/tests/session_protocol.rs @@ -3092,3 +3092,71 @@ fn expert_voice_speaks_replies_itself_and_tells_the_spokesperson_what_was_heard( session.shutdown(); server.join().unwrap(); } + +#[test] +fn expert_voice_reply_waits_for_output_ready_before_playing() { + let (speech_url, speech_requests) = spawn_speech_server(); + let (endpoint_tx, endpoint_rx) = mpsc::sync_channel(1); + std::thread::spawn(move || { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap() + .block_on(async move { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + endpoint_tx + .send(format!("ws://{}/", listener.local_addr().unwrap())) + .unwrap(); + let (stream, _) = listener.accept().await.unwrap(); + let mut socket = accept_async(stream).await.unwrap(); + let initial = receive_realtime_json(&mut socket).await; + acknowledge_realtime_session(&mut socket, &initial).await; + while socket.next().await.is_some() {} + }); + }); + let mut session = ExpertSpokespersonTestSession::start_with_args( + endpoint_rx.recv().unwrap(), + &["--expert-voice", "tts", "--tts-url", &speech_url], + ); + session.send(json!({ + "type":"prepare_speak", + "id":2, + "acknowledgement":null, + "text":"I checked the repository." + })); + let admitted = session.recv(Duration::from_secs(2)); + assert_eq!(admitted["type"], "admitted", "{admitted}"); + let speech_id = admitted["speech_id"].as_u64().unwrap(); + if let Ok(message) = session.output.recv_timeout(Duration::from_millis(500)) { + panic!("the reply went on before output_ready: {message}"); + } + assert!( + speech_requests.try_recv().is_err(), + "the reply was synthesized before output_ready" + ); + for (id, speech_id) in [(2, speech_id + 1), (3, speech_id)] { + session.send(json!({"type":"output_ready","id":id,"speech_id":speech_id})); + assert_eq!( + session.recv(Duration::from_secs(2)), + json!({"type":"output_ready_result","id":id,"speech_id":speech_id,"outcome":"stale"}) + ); + } + session.send(json!({"type":"output_ready","id":2,"speech_id":speech_id})); + assert_eq!( + session.recv(Duration::from_secs(2)), + json!({"type":"output_ready_result","id":2,"speech_id":speech_id,"outcome":"accepted"}) + ); + let mut started = false; + loop { + let message = session.recv(Duration::from_secs(3)); + match message["type"].as_str() { + Some("speech_started") => started = true, + Some("speech_completed") => break, + _ => panic!("unexpected message: {message}"), + } + } + assert!(started); + session.send(json!({"type":"output_ready","id":2,"speech_id":speech_id})); + assert_eq!(session.recv(Duration::from_secs(2))["outcome"], "stale"); + session.shutdown(); +} From ef2eadd5c70daa27d55c9b23c5de4972516ed7d3 Mon Sep 17 00:00:00 2001 From: John Tennant Date: Tue, 20 Oct 2026 13:15:12 -0400 Subject: [PATCH 10/10] Tell the Spokesperson what was heard of a failed Expert reply A failed Expert reply was treated as unheard, so its handoffs were released and the Spokesperson was told the user heard none of it. Playback failures now carry what had been played, so the heard prefix reaches the Spokesperson and an answered handoff is dismissed. Signed-off-by: John Tennant Co-Authored-By: Claude Opus 5.5 --- src-tauri/crates/berd-call/src/main.rs | 26 ++-- .../berd-call/tests/session_protocol.rs | 123 ++++++++++++++++++ 2 files changed, 141 insertions(+), 8 deletions(-) diff --git a/src-tauri/crates/berd-call/src/main.rs b/src-tauri/crates/berd-call/src/main.rs index 1cd81fce9..ed85acaf7 100644 --- a/src-tauri/crates/berd-call/src/main.rs +++ b/src-tauri/crates/berd-call/src/main.rs @@ -138,6 +138,8 @@ enum PlaybackEvent { #[derive(Debug)] struct PlaybackFailure { message: String, + /// What the host had played when playback failed. + delivery: DeliveryProgress, output_quiescent: bool, } @@ -4190,7 +4192,7 @@ fn run_expert_spokesperson_session( speech_id, message: failure.message, }, - 0, + estimated_spoken_through_utf8(&text, &failure.delivery), ) } }; @@ -8172,17 +8174,16 @@ fn play_tts( active: &AtomicBool, ) -> Result<(bool, DeliveryProgress), PlaybackFailure> { if let Err(message) = output.start() { + let delivery = DeliveryProgress { + sample_rate: backend.pcm_spec().sample_rate, + segments: Vec::new(), + }; if message == AUDIO_CANCELLED { - return Ok(( - false, - DeliveryProgress { - sample_rate: backend.pcm_spec().sample_rate, - segments: Vec::new(), - }, - )); + return Ok((false, delivery)); } return Err(PlaybackFailure { message, + delivery, output_quiescent: output.failure_is_quiescent(), }); } @@ -8223,19 +8224,26 @@ fn synthesize_to_output_with_finish( use berd_call::{DrainPolicy, DrainTimeoutOutcome, OutboundOutcome, OutboundPlayback}; let spec = backend.pcm_spec(); + let unplayed = || DeliveryProgress { + sample_rate: spec.sample_rate, + segments: Vec::new(), + }; let initial_frames = usize::try_from(spec.sample_rate / 5).map_err(|_| PlaybackFailure { message: "TTS sample rate is too large".into(), + delivery: unplayed(), output_quiescent: false, })?; let mut playback = OutboundPlayback::new(output, active, spec.sample_rate, initial_frames) .map_err(|message| PlaybackFailure { message, + delivery: unplayed(), output_quiescent: false, })?; if playback .synthesize_segment(backend, text, &mut |_| Ok(()), on_started, &mut |_| Ok(())) .map_err(|failure| PlaybackFailure { message: failure.message, + delivery: failure.delivery, output_quiescent: failure.output_quiescent, })? == OutboundOutcome::Interrupted @@ -8246,6 +8254,7 @@ fn synthesize_to_output_with_finish( let output_quiescent = output.cancel_and_snapshot().is_ok(); return Err(PlaybackFailure { message, + delivery: playback.snapshot(), output_quiescent, }); } @@ -8260,6 +8269,7 @@ fn synthesize_to_output_with_finish( ) .map_err(|failure| PlaybackFailure { message: failure.message, + delivery: failure.delivery, output_quiescent: failure.output_quiescent, })?; Ok((outcome == OutboundOutcome::Completed, playback.snapshot())) diff --git a/src-tauri/crates/berd-call/tests/session_protocol.rs b/src-tauri/crates/berd-call/tests/session_protocol.rs index 090b3d2c2..dd2da7a78 100644 --- a/src-tauri/crates/berd-call/tests/session_protocol.rs +++ b/src-tauri/crates/berd-call/tests/session_protocol.rs @@ -3160,3 +3160,126 @@ fn expert_voice_reply_waits_for_output_ready_before_playing() { assert_eq!(session.recv(Duration::from_secs(2))["outcome"], "stale"); session.shutdown(); } + +/// Fails an Expert reply answering a handoff once the host has played `played_frames` of it, +/// returning the failure, what the Spokesperson was told, and the handoffs left open. +fn fail_expert_reply_after(played_frames: u64) -> (Value, String, Value) { + let (speech_url, _speech_requests) = spawn_speech_server(); + let (endpoint_tx, endpoint_rx) = mpsc::sync_channel(1); + let (context_tx, context_rx) = mpsc::sync_channel(1); + std::thread::spawn(move || { + tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + .unwrap() + .block_on(async move { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + endpoint_tx + .send(format!("ws://{}/", listener.local_addr().unwrap())) + .unwrap(); + let (stream, _) = listener.accept().await.unwrap(); + let mut socket = accept_async(stream).await.unwrap(); + let initial = receive_realtime_json(&mut socket).await; + acknowledge_realtime_session(&mut socket, &initial).await; + send_realtime_json( + &mut socket, + json!({ + "type":"response.function_call_arguments.done", + "response_id":"response-handoff", + "call_id":"handoff-reply", + "name":"handoff", + "arguments":"{\"message\":\"inspect the repository\"}" + }), + ) + .await; + let context = loop { + let text = receive_realtime_json(&mut socket).await.to_string(); + if text.contains("spoken reply") || text.contains("in its own voice") { + break text; + } + }; + context_tx.send(context).unwrap(); + while socket.next().await.is_some() {} + }); + }); + let (observed, playing) = mpsc::sync_channel(1); + let (release_drain, release) = mpsc::sync_channel(1); + let mut session = ExpertSpokespersonTestSession::launch( + endpoint_rx.recv().unwrap(), + None, + Some(played_frames), + None, + Some(AudioCancellationGate { + observed, + release, + at_drain: true, + }), + &["--expert-voice", "tts", "--tts-url", &speech_url], + ); + assert_eq!(session.recv(Duration::from_secs(2))["origin"], "handoff"); + assert_eq!( + session.recv(Duration::from_secs(2))["handoff_ids"], + json!(["handoff-external-1"]) + ); + session.send(json!({ + "type":"prepare_speak", + "id":2, + "acknowledgement":1, + "text":"I checked the repository and found three failing tests in the parser.", + "resolved_handoff_ids":["handoff-external-1"] + })); + let admitted = session.recv(Duration::from_secs(2)); + assert_eq!(admitted["type"], "admitted", "{admitted}"); + session.send(json!({"type":"output_ready","id":2,"speech_id":admitted["speech_id"]})); + playing.recv_timeout(Duration::from_secs(2)).unwrap(); + release_drain.send(()).unwrap(); + session.send(json!({ + "type":"audio_failed", + "speech_id":admitted["speech_id"], + "played_frames":played_frames, + "message":"output device disappeared" + })); + let failed = loop { + let message = session.recv(Duration::from_secs(2)); + assert!( + !matches!( + message["type"].as_str(), + Some("fatal" | "speech_completed" | "speech_interrupted") + ), + "{message}" + ); + if message["type"] == "speech_failed" { + break message; + } + }; + let context = context_rx.recv_timeout(Duration::from_secs(2)).unwrap(); + session.send(json!({"type":"query_state","id":3,"after":0})); + let state = loop { + let message = session.recv(Duration::from_secs(2)); + if message["type"] == "state" { + break message; + } + }; + session.shutdown(); + (failed, context, state["unresolved_handoff_ids"].clone()) +} + +#[test] +fn a_failed_expert_reply_tells_the_spokesperson_what_was_heard() { + let (failed, context, unresolved) = fail_expert_reply_after(1_000); + assert_eq!(failed["id"], 2); + assert!( + context.contains("in its own voice") && context.contains("I checked the repository"), + "{context}" + ); + assert!(!context.contains("in the parser"), "{context}"); + assert!(unresolved.is_null(), "{unresolved}"); +} + +#[test] +fn an_expert_reply_that_fails_unheard_leaves_its_handoffs_open() { + let (failed, context, unresolved) = fail_expert_reply_after(0); + assert_eq!(failed["id"], 2); + assert!(context.contains("before hearing any of it"), "{context}"); + assert_eq!(unresolved, json!(["handoff-external-1"])); +}