berd-call: realtime calls for agents: live stream, own voices, and Spokesperson controls

src-tauri/crates/berd-call/PROTOCOL.md

@@ -72,6 +72,14 @@ OpenAI snapshots contain `model`, `voice`, and `rate`; Siri contains `voice`,
 and `rate`. Credentials, endpoints, and bundle paths never appear on stdout.
 Detailed backend errors are diagnostics on stderr only; protocol rejection and
 fatal messages are sanitized at the stdout boundary.
+`session.tts` describes the voice that speaks replies, and `set_tts_settings`
+changes it: in Expert-Spokesperson sessions, that is the configured TTS backend
+with `--allow-direct-agent-speech` and otherwise the Spokesperson's own voice. With
+`--allow-direct-agent-speech`, `session.spokesperson` is the Spokesperson's own OpenAI
+snapshot, with its own revision, and `set_spokesperson_settings` changes it the
+way `set_tts_settings` changes the Spokesperson's voice in other sessions. A reply in
+the Expert's own voice reports the same `admitted`, `speech_completed`,
+`speech_interrupted`, and `speech_failed` lifecycle as a conventional reply.
 `session.input_during_tts` is the authoritative effective assistant-input
 policy and has its own revision. Status-sound settings are per-update parameters,
 not session snapshot configuration. The runtime owns cue cadence and playback; it
@@ -195,9 +203,10 @@ it never admits a replacement while old host audio may still be active.
 {"type":"set_input_muted","id":u64,"active":bool}
 {"type":"set_conversation_status","id":u64,"status":"working"|"waiting","settings":StatusSoundSettings}
 {"type":"set_tts_settings","id":u64,"expected_revision":u64,"settings":TtsSettings}
+{"type":"set_spokesperson_settings","id":u64,"expected_revision":u64,"settings":TtsSettings}
 {"type":"set_input_during_tts","id":u64,"expected_revision":u64,"policy":"allow_barge_in"|"suppress_input"}
 {"type":"reset_input","id":u64}
-{"type":"prepare_speak","id":u64,"acknowledgement":u64|null,"text":string}
+{"type":"prepare_speak","id":u64,"acknowledgement":u64|null,"text":string,"resolved_handoff_ids":[string]?,"voice":string?,"language":string?,"relay":bool?}
 {"type":"output_ready","id":u64,"speech_id":u64}
 audio acknowledgements listed above
 {"type":"query_state","id":u64,"after":u64}
@@ -207,6 +216,19 @@ audio acknowledgements listed above
 ```
 
 Unknown fields are rejected. IDs are positive. Speak text is at most 16 KiB.
+`prepare_speak.voice` speaks that one utterance in another voice of the reply
+backend, keeping its other settings; `language` changes a Siri voice's locale
+too. A voice the backend doesn't offer, or a voice for a call whose replies
+are spoken by the Spokesperson, is `not_admitted(invalid_voice)`. In a call
+whose replies have their own voice, a reply needs the same `acknowledgement`
+as a directive. With `relay: true` (what
+`berd-call message-spokesperson` sends), the reply is a directive in any
+Expert-Spokesperson call, even one whose replies have their own voice: the
+Spokesperson says it, on the same `acknowledgement` and `resolved_handoff_ids`
+terms. A call whose replies have their own voice holds no handoff open: the
+Expert still hears each one, but `unresolved_handoff_ids` stays empty, no
+reminder or renewal waits on them, and naming one in `resolved_handoff_ids` is
+`not_admitted(invalid_handoff)`.
 The parent cannot author speaking state or finalized input; those are derived
 only from PCM by the child runtime.
 
@@ -241,6 +263,11 @@ linearization point. A speech reservation holds a configuration lease: speech
 admitted before the response retains its old backend/settings, while later
 admission receives the new revision. Pocket's public model identifier cannot be
 changed without selecting and validating another bundle at process startup.
+In a `berd-call start` realtime call, a Spokesperson voice or rate change that
+the conversation overtakes while its new session is being built is not
+rejected: the candidate is discarded and the change is tried again, with the
+conversation as it then stands, once the call is idle. Its one result comes
+when it finally applies or fails.
 
 `set_input_during_tts` changes the policy for later admissions. Its expected
 revision must be positive. The correlated result is always authoritative:
@@ -281,6 +308,21 @@ Expert-Spokesperson sessions use `origin` to distinguish user transcripts,
 Spokesperson transcripts, and handoffs. User finals interrupt reserved or playing
 assistant output only after storage and emission. Final text is at most 64 KiB.
 
+Expert-Spokesperson sessions also emit two previews for hosts that show the
+conversation as it happens:
+
+```text
+{"type":"user_transcribed","token":u64,"text":string}
+{"type":"spokesperson_generated","text":string}
+```
+
+`user_transcribed` follows a user final's `live_event` with its token and plain
+text. `spokesperson_generated` carries a reply's whole text once the provider
+finishes generating it, before playback, and only after no user speech is still
+being transcribed. A reply already spoken or cut off by then gets none. The
+Spokesperson transcript recorded after playback stays authoritative. Neither
+message changes Expert delivery.
+
 ## Confirmation and admission
 
 `acknowledgement` is a request-local causal cutoff. `null` uses the stored
@@ -324,7 +366,7 @@ cancellation and shutdown terminals.
 
 ```text
 {"type":"pending","id":u64,"utterances":[{"token":u64,"text":string}]}
-{"type":"not_admitted","id":u64,"reason":"paused"|"in_progress"|"cancelled"|"empty_text"}
+{"type":"not_admitted","id":u64,"reason":"paused"|"in_progress"|"cancelled"|"empty_text"|"invalid_handoff"|"invalid_voice"}
 {"type":"state","id":u64,"confirmed_token":u64,"utterances_after":[{"token":u64,"text":string}]}
 {"type":"cancel_result","id":u64,"outcome":"cancelled"|"stale","speech_id":u64|null}
 {"type":"speech_started","id":u64,"speech_id":u64}

src-tauri/crates/berd-call/README.md

@@ -57,14 +57,25 @@ Start a call with the same backend flags accepted by `session`:
 berd-call start --voice Aaron --language en-US --stream
 berd-call status
 berd-call speak --re 7 "I found the cause."
+berd-call speak --voice Nora "Sol here: the tests pass."
 berd-call stop
 ```
 
+`speak --voice` speaks one reply in another voice of the call's reply backend
+without changing the call's voice, so agents sharing a call can sound distinct.
+
 `start` stays in the foreground and owns the child session process. The control
 server listens only on `127.0.0.1:5222` by default; all four commands accept
 `--port` when more than one local call must be addressed. `--stream` writes one
 TSV row per delivered transcript event after the `cursor`, `role`, and `text`
-header. The standalone host intentionally uses only the default devices. It
+header. In Expert-Spokesperson mode the stream hears the call as Berd's Expert
+does: a batch of every row since the last one after each Spokesperson reply,
+and at each handoff, ending with the `handoff` row. `--stream live` instead shows the conversation as it happens: a
+`user` row as soon as the words are transcribed, and a `spokesperson_generated`
+row with cursor `0` as soon as a reply is generated, before it is spoken. The
+`spokesperson` or `spokesperson_interrupted` row after playback carries the
+reply's cursor and what was heard. A conventional call has no handoffs, so it
+always streams each row. A `--codex` task gets the same batches as `--stream`. The standalone host intentionally uses only the default devices. It
 suppresses recognition during playback when the default output
 identifies itself as built-in speakers, preventing the call from transcribing
 its own speech; headphone-like outputs retain barge-in.
@@ -98,6 +109,11 @@ complete `berd-call session` options. The call keeps its control port, stream,
 blocking policy, mute state, and input-during-TTS policy; the stream emits a lifecycle row because
 transcript cursors restart with the new session.
 
+`berd-call settings --spokesperson true|false`, or Spokesperson in the menu bar,
+restarts the call the same way in the other mode. Agents keep their current
+voice, options only the other mode takes are left out, and in a `--stream live`
+call the Spokesperson comes back with `--allow-direct-agent-speech`.
+
 This crate owns the neutral PCM output contract and backend-neutral TTS stream
 used by Berd, plus the April ONNX runtime and text chunking used by Berd's native
 voice commands. It also owns the concrete Parakeet model loader and complete
@@ -133,11 +149,14 @@ berd-call session --tts-backend pocket --model-dir /path/to/native-voice-v2 --vo
 berd-call session --stt-backend parakeet --stt-model-dir /path/to/parakeet
 berd-call session --stt-backend openai
 berd-call session --tts-backend openai --mode expert-spokesperson --realtime-url ws://127.0.0.1:18870/v1/realtime
+berd-call session --voice Aaron --language en-US --mode expert-spokesperson --allow-direct-agent-speech
 berd-call session --tts-backend openai --tts-url https://proxy.example/v1/audio/speech
 berd-call session --stt-backend openai --stt-url wss://proxy.example/v1/realtime?intent=transcription
 ```
 
-The three URL flags take full endpoints and override only their matching service for that call. Expert–Spokesperson uses `--realtime-url` for both input and output; `--stt-url` and `--tts-url` apply only to conventional mode. Without an override, each service uses its OpenAI endpoint (or its existing environment override). `OPENAI_API_KEY` supplies the CLI credential; the desktop app stores keys separately by endpoint URL in Keychain.
+In an Expert–Spokesperson call, the Expert reaches you through the Spokesperson: `berd-call message-spokesperson --re CURSOR [--resolves HANDOFF_ID]... TEXT` has it tell you in its own voice. Like the directives of Berd's Expert, the message must acknowledge the latest input (the Spokesperson's own speech doesn't count), or it comes back `spoke:false` with what was missed, and a handoff it names settles once the Spokesperson has said it. `speak` is refused unless the call was started with `--allow-direct-agent-speech`. With it, the Expert can also speak for itself: berd-call plays each `speak` reply through the configured TTS backend, taking turns with the Spokesperson's audio, and your speech interrupts either of them. Afterwards the Spokesperson gets a silent note of what you heard, so it can follow up without repeating it. A reply has to acknowledge everything you've said, as `speak --re` always does. Such a call holds no handoff open: each still reaches the Expert as a `handoff` row, but nothing waits for an answer, reminds the Expert, or holds back renewing or recovering the Spokesperson's session, and there is nothing for `--resolves` to name. `--voice`, `--rate`, `status` and `settings --tts` describe and change the Expert's voice. The Spokesperson keeps its own voice, shown as `session.spokesperson`: choose it with `--spokesperson-voice NAME`, at `start` or live with `settings`, and the menu bar offers both voices. It speaks as close to the agents' `--rate` as it can (0.25–1.5×), so the menu bar's one Speech Rate sets both; `--spokesperson-rate` gives it a speed of its own. Saved settings keep the Spokesperson's voice for the next call whose replies have their own, but not `--allow-direct-agent-speech`, so pass that on each `start`.
+
+The three URL flags take full endpoints and override only their matching service for that call. Expert–Spokesperson uses `--realtime-url` for both input and output; `--stt-url` applies only to conventional mode, and `--tts-url` also applies to the Expert's own voice. Without an override, each service uses its OpenAI endpoint (or its existing environment override). `OPENAI_API_KEY` supplies the CLI credential; the desktop app stores keys separately by endpoint URL in Keychain.
 
 The default Siri backend still requires an exact installed voice name and
 language. Missing or unavailable Siri voice configuration and an unavailable
@@ -254,7 +273,9 @@ retries cleanup. This avoids reporting a failed install after the model has
 already been applied.
 
 `voices.list` returns `backend`, normalized `languageFilter`,
-`availableLanguages`, and exact voice records. `voices.download` returns the
+`availableLanguages`, and exact voice records. A voice is `installed` only when
+sirittsd reports it downloaded and its assets are on disk: sirittsd can claim
+a voice whose assets are missing, then silently speak it in another voice. `voices.download` returns the
 canonical voice, `installed: true`, and `availabilityWaitSeconds`; a missing
 exact catalog identity fails with `voice_not_found` before any native download
 request. macOS status and install results contain `supported`, `locale`,

src-tauri/crates/berd-call/native/siri_tts_bridge.m

@@ -997,10 +997,14 @@ static void BerdDownloadedVoices(
         NSString *name = [resource valueForKey:@"name"] ?: @"";
         if (!name.length || !language.length) continue;
         NSString *key = [NSString stringWithFormat:@"%@|%@", name, normalizedLanguage];
+        // sirittsd can report a voice as downloaded whose assets aren't on disk,
+        // and then speaks it in another voice.
+        NSString *content = [resource valueForKey:@"contentPath"];
         byKey[key] = @{
             @"name" : name,
             @"language" : language,
             @"sizeBytes" : [resource valueForKey:@"assetSize"] ?: @0,
+            @"content" : @(content.length && [NSFileManager.defaultManager fileExistsAtPath:content]),
         };
     }
     return [[byKey allValues] sortedArrayUsingComparator:
@@ -1142,7 +1146,9 @@ static BOOL BerdTriggerDownload(NSString *language, NSError **error) {
             NSString *key = [NSString stringWithFormat:@"%@|%@",
                 candidate[@"name"], normalizedLanguage];
             NSMutableDictionary *voice = [candidate mutableCopy];
-            voice[@"installed"] = installedSnapshot[key] ?: @NO;
+            [voice removeObjectForKey:@"content"];
+            voice[@"installed"] =
+                [installedSnapshot[key] boolValue] && [candidate[@"content"] boolValue] ? @YES : @NO;
             [result addObject:voice];
         }
         NSData *json = [NSJSONSerialization dataWithJSONObject:result options:0 error:&error];