Skip to content

Speak with Chatterbox and Pocket TTS, with predefined voices - #1792

Merged
leehack merged 8 commits into
mainfrom
feat/tts-chatterbox-pocket
Nov 8, 2026
Merged

leehack merged 8 commits into
mainfrom
feat/tts-chatterbox-pocket

Conversation

@leehack

@leehack leehack commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

Production-readiness scope

  • User-facing scope: Chatterbox Turbo and Multilingual speech, voice default or speaker-reference cloning, Multilingual language; Pocket TTS speech with 25 predefined voices. The example CLI gains the two adapters plus --speaker and --sequences.
  • Supported platforms/paths: Native llama.cpp runtimes that report types 4/5 in llama_dart_tts_supported_model_types (llamadart-native v0.9.0 or later).
  • Unsupported or intentionally unavailable paths:
    • v0.8.0 and older, or Apple companion locks on them: load throws LlamaUnsupportedException naming the symbol and v0.9.0.
    • Chatterbox Multilingual in a context with fewer than two sequences: capabilities report unsupported, asking for ModelParams.maxParallelSequences of at least 2 (the classifier-free-guidance twin sequence).
    • Voice plus speaker reference: LlamaTextToSpeechException before synthesis. Turbo and Pocket TTS: language throws LlamaUnsupportedException. Pocket without a voice encoder: speaker reference throws LlamaUnsupportedException.
    • Web/WebGPU: load refuses these families (Qwen3-TTS only).
    • Adreno 750 Vulkan stays off through the existing Android driver guard.
  • Out of scope / follow-ups: None.

Completeness checklist

  • Declared scope is fully implemented, or explicitly reduced above.
  • Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable.
  • Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant.
  • Regression coverage covers the original issue plus key negative/version-skew paths where relevant.
  • Public capability changes update the validation coverage registry with a pack or a reasoned, issue-linked exemption; inventory checks pass (or N/A with scope). No new capability fields; the families are covered by the text-to-speech-models-smoke exemption reason from PR1.
  • Security/privacy review completed: no secrets, bearer tokens, signed URLs, or raw secret-bearing paths leak through logs, cache keys, metadata, errors, or snapshots.
  • Follow-up work that is useful but not required for this PR is tracked in GitHub Issues, or explicitly marked "None" above.

PR type guidance

  • Feature PR: API, docs, example, support matrix, unsupported paths and positive/negative tests are included above.
  • Bugfix PR: N/A, feature.
  • Docs-only PR: N/A, runtime change.

Test Plan

  • dart run tool/prepare_workspace.dart
  • dart format --output=none --set-exit-if-changed .
  • dart analyze
  • dart test -p vm -j 1 --exclude-tags local-only: full suite 11518 pass, 129 skip, 6 fail (all windows_registry_qualification_test.dart: no pwsh on this host) at 672b0de; at this head the TTS, worker, backend, entrypoint, structure and tooling suites pass (692 tests), and PR CI runs the full suite.
  • dart test -p chrome --exclude-tags local-only: full suite 4626 pass, 1 skip before the retarget; at this head the Web, WebGPU and speech suites pass (391 tests), and PR CI runs the full suite.
  • coverage-lib matrix row: not run locally; main enforces it after merge.
  • Other targeted/local-only validation: text-to-speech-models-smoke, llama-cpp-exit-teardown (quit-speaking), version skew on v0.8.0-3.

Matrix Evidence

Matrix row Scope covered Platform / model / backend Result Evidence / notes
text-to-speech-models-smoke (published v0.9.0, this head) Chatterbox Turbo, Multilingual, Pocket TTS after merging main with #1791 macOS arm64 M4 Max, published llamadart-native v0.9.0 (hook output sha256 9d9cdbc6…), no local override, CPU, 2/2 threads, isolated TMPDIR Pass (20 pass, 25 skip: LFM3/Soprano unset plus inapplicable cases) Transcripts exact for Turbo default and cloned, Multilingual default and cloned, Pocket default and vera; Multilingual fr gives "Bonjour, comment allez-vous aujourd'hui?".
text-to-speech-models-smoke (published v0.9.0, all families) All five families after merging main (#1784 landed, pin v0.9.0); no local native override macOS arm64 M4 Max, published llamadart-native v0.9.0 bundle (hook output sha256 9d9cdbc6…), CPU, 2/2 threads, isolated TMPDIR Pass (29 pass, 16 inapplicable skips) ASR transcripts exact for LFM3 default and male, Soprano, Chatterbox Turbo default and cloned, Multilingual default and cloned, Pocket default and vera; Multilingual fr gives "Bonjour, comment allez-vous aujourd'hui?". Run at 672b0de, before the retarget.
text-to-speech-models-smoke (pre-release native) Chatterbox Turbo, Multilingual, Pocket TTS: capabilities, voices, cloning, language, refusals, seed repeat after cancel, Qwen3-ASR round trip macOS arm64 M4 Max, local llamadart-native main bee747a5, CPU, 2/2 threads; ggml-org/chatterbox-turbo-GGUF Q8_0, ggml-org/chatterbox-GGUF Q8_0 (F16 mmprojs), macolomb/pocket-tts-without-voice-cloning english; ASR ggml-org/Qwen3-ASR-0.6B-GGUF Q8_0 Pass (20 pass, 25 skip: LFM3/Soprano unset in that run plus inapplicable cases) Transcripts exact for Turbo default and cloned, Multilingual default and cloned, Pocket default and vera; Multilingual fr gives "Bonjour, comment allez-vous aujourd'hui?". Pocket lists 25 voices; no voice equals the first voice; Multilingual refuses a one-sequence context.
version skew: v0.8.0-3 pre-load refusal pinned v0.8.0-3, example CLI Pass Chatterbox and Pocket TTS throw LlamaUnsupportedException naming v0.9.0 before loading; LFM3/Soprano still speak.
llama-cpp-exit-teardown (quit-speaking) exit() mid-synthesis macOS arm64, local llamadart-native main bee747a5 (pre-release), Metal Pass {"quit-speaking":[0,0,0]} for Chatterbox Multilingual and Pocket TTS.
probe mutation checks worker, backend client, Web and WebGPU answers to the pre-load probe VM and Chrome Pass Nulling the worker's reason, the backend client's return, either Web backend's override, or refusing Qwen3 in the shared Web function each fails the matching new test.
model-free integration type mask 0x3e, Multilingual sequence refusal, Turbo/Pocket capability flags, version messages VM, mirror-injected fake _TtsApi Pass test/integration/backends/llama_cpp_tts_model_types_test.dart, text_to_speech_model_test.dart.
llama-cpp-host-shutdown (Linux) N/A - Not run No Linux host.
chat-app-macos-quit N/A - Not run Chat app TTS is Qwen3-only and unchanged.
coverage-lib N/A - Not run Enforced on main after merge.
Windows / Linux / Android / iOS real-model N/A - Not run No such hosts or devices here.

Review Notes

High-risk regression review

  • Classification: high-risk (backendRuntime, regressionPolicy).
  • Implementation task: this PR's author session.
  • Independent blocking QA task: done, APPROVE; VM 410 targeted and 3392 broad (1 unrelated LiteRT flake under -j 4), Chrome 383, coverage and skills 48.
  • Exact head / current base: 99891bf160a1f2dbe7ab9de00812d128942495b6 / ce8ddf674200297e5883d74e8888552c4af612b1.
  • Production-branch deletion, bypass, or miswire proof: auditor killed 17 of 17 mutants (multilingual gate, version map, adapter supportsModel, probe routing through worker, backend, _runtimeRefusal, Web, WebAutoBackend, WebGpu); real v0.8.0 refuses Chatterbox and Pocket before load. The integration suite fails if the type mask drops types 4/5 or the Multilingual sequence check is bypassed.
  • Affected-family real-model/artifact evidence: Chatterbox Turbo, Chatterbox Multilingual, Pocket TTS, see Matrix Evidence.
  • Explicit unavailable-family or other N/A evidence: Pocket TTS cloning with a voice-encoder GGUF is not exercised: the GGUF used ships no encoder, and the E2E asserts its speaker-reference refusal. Dart reads the native SPEAKER_REFERENCE bit, so no family-specific code path is untested.
  • Known PR-caused P1 regressions: 0
  • Unresolved review threads: 0
  • Audit P3s (not blocking): the exit-probe fixture now loads every adapter with maxParallelSequences: 2; Soprano was not run on real weights.

Generated from llamadart-native main eb760895 (llama.cpp cd959bae) headers,
packaged as the release workflow does, with native_header_archive.py and
ffigen in a Linux container. The same procedure reproduces the committed
v0.8.0-3 bindings byte for byte.
…type opt-in

Drive llama_dart_tts_get_info_ext/init_ext/start_with_voice with the mask of
types this core handles and the runtime builds, list predefined voices when
the runtime exports the voice API, and add Lfm3AudioTtsAdapter,
SopranoTtsAdapter, TextToSpeechRequest.voice and
TextToSpeechCapabilities.supportsVoice/voices. TextToSpeechEngine.load asks
the backend which models its runtime drives and refuses an adapter none of
whose models it drives before loading anything, naming the required
llamadart-native version.

Refs #1615.
Chatterbox Turbo and Multilingual and Pocket TTS speak through the native
model-type opt-in once llamadart-native v0.9.0 reports types 4 and 5.
Pocket TTS lists its projector's voices; Chatterbox takes `default` or a
speaker reference, and Multilingual a language and a two-sequence context.
Older runtimes refuse before loading, naming v0.9.0.

Refs #1783.
…rano

# Conflicts:
#	website/docs/platforms/support-matrix.md
…-pocket

# Conflicts:
#	website/docs/platforms/support-matrix.md
…b backends

The llama.cpp worker, its backend client, and the Web and WebGPU backends
each answer the pre-load probe; nulling any of their answers now fails a
test.

Refs #1615.
Base automatically changed from feat/tts-lfm3-soprano to main November 8, 2026 13:07
…ox-pocket

# Conflicts:
#	README.md
#	doc/testing_matrix.md
#	example/basic_app/README.md
#	example/basic_app/bin/llamadart_text_to_speech_example.dart
#	lib/src/backends/backend.dart
#	lib/src/backends/llama_cpp/llama_cpp_service.dart
#	lib/src/core/speech/text_to_speech_model.dart
#	skills/llamadart-speech/SKILL.md
#	test/e2e/backends/llama_cpp_exit_teardown_e2e_test.dart
#	test/e2e/backends/text_to_speech_models_e2e_test.dart
#	test/fixtures/llama_cpp_exit_probe.dart
#	test/integration/backends/llama_cpp_tts_model_types_test.dart
#	test/unit/core/speech/text_to_speech_model_test.dart
#	tool/testing/run_local_e2e.dart
#	tool/testing/test_matrix.dart
#	website/docs/getting-started/model-families.md
#	website/docs/guides/text-to-speech.md
#	website/docs/platforms/support-matrix.md
@leehack
leehack marked this pull request as ready for review November 8, 2026 14:10
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Chat app preview removed for leehack/llamadart-chat-pr-1792.

@leehack
leehack merged commit d53971e into main Nov 8, 2026
46 checks passed
@leehack
leehack deleted the feat/tts-chatterbox-pocket branch November 8, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant