Skip to content

Speak with LFM3-Audio and Soprano through the native model-type opt-in - #1791

Merged
leehack merged 4 commits into
mainfrom
feat/tts-lfm3-soprano
Nov 8, 2026
Merged

leehack merged 4 commits into
mainfrom
feat/tts-lfm3-soprano

Conversation

@leehack

@leehack leehack commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Speak with LFM3-Audio and Soprano through the guarded native _ext transport (llama_dart_tts_supported_model_types, get_info_ext, init_ext, start_with_voice), selected by runtime probe, never by version. Refs Expose LFM3 and Soprano speech output from native v0.8.0 #1615.
  • Add predefined voices: TextToSpeechRequest.voice, TextToSpeechCapabilities.supportsVoice/voices, read from llama_dart_tts_get_voice_count/get_voice_name when the runtime lists them. The voice API lands here, not in the Expose Chatterbox and Pocket TTS voices from native v0.9.0 #1783 PR, because LFM3-Audio already takes female/male.
  • TextToSpeechEngine.load asks the runtime which families it drives before loading any file (new backend SPI BackendTextToSpeechModelSupport), so a runtime that cannot load the LFM3/Soprano architecture refuses with a typed LlamaUnsupportedException naming the missing symbol and llamadart-native v0.8.0, instead of a generic model-load failure.
  • Targets main, which now pins llamadart-native v0.9.0 (Adopt llamadart-native v0.9.0 #1784 merged; main merged into this branch). Chatterbox and Pocket TTS follow in Speak with Chatterbox and Pocket TTS, with predefined voices #1792, stacked on this one.

Production-readiness scope

  • User-facing scope: Lfm3AudioTtsAdapter and SopranoTtsAdapter for TextToSpeechEngine; LFM3 voices female (default) and male plus pass-through language; Soprano 32 kHz output. New CLI example/basic_app/bin/llamadart_text_to_speech_example.dart.
  • Supported platforms/paths: Native llama.cpp runtimes (VM, Flutter native) that export the opt-in symbols: llamadart-native v0.8.0 or later for synthesis; voice listing from v0.9.0.
  • Unsupported or intentionally unavailable paths:
    • Runtimes without the opt-in (v0.7.x, older Apple companion locks): load throws LlamaUnsupportedException naming llama_dart_tts_init_ext and v0.8.0; Qwen3-TTS keeps working.
    • v0.8.0 (no voice listing): voices is empty; a requested voice is still passed to start_with_voice, and an unknown one fails with the native "unknown TTS voice" error.
    • LFM3 GGUF without its tts chat template: capabilities report unsupported with that reason.
    • Web/WebGPU: load refuses non-Qwen3 families ("Web text-to-speech drives Qwen3-TTS only").
    • Soprano: voice, language and speaker reference throw LlamaUnsupportedException before synthesis.
    • Adreno 750 Vulkan stays off: the existing Android driver guard keeps GPU work on CPU for these models (documented in the guide).
  • Out of scope / follow-ups: Chatterbox Turbo/Multilingual and Pocket TTS (Expose Chatterbox and Pocket TTS voices from native v0.9.0 #1783, stacked PR). Chat app TTS stays Qwen3-only.

Completeness checklist

  • Declared scope is fully implemented, or explicitly reduced above.
  • Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable.
  • Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant.
  • Regression coverage covers the original issue plus key negative/version-skew paths where relevant.
  • Public capability changes update the validation coverage registry with a pack or a reasoned, issue-linked exemption; inventory checks pass (or N/A with scope). supportsVoice/voices join the speech-capabilities exemption, whose reason now cites text-to-speech-models-smoke and Expose LFM3 and Soprano speech output from native v0.8.0 #1615.
  • Security/privacy review completed: no secrets, bearer tokens, signed URLs, or raw secret-bearing paths leak through logs, cache keys, metadata, errors, or snapshots.
  • Follow-up work that is useful but not required for this PR is tracked in GitHub Issues, or explicitly marked "None" above.

PR type guidance

  • Feature PR: API, docs, example, support matrix, unsupported paths and positive/negative tests are included above.
  • Bugfix PR: N/A, feature.
  • Docs-only PR: N/A, runtime change.

Test Plan

  • dart run tool/prepare_workspace.dart
  • dart format --output=none --set-exit-if-changed .
  • dart analyze
  • dart test -p vm -j 1 --exclude-tags local-only (macOS arm64, after merging main with its v0.9.0 pin, isolated TMPDIR): 11514 pass, 129 skip, 6 fail (all windows_registry_qualification_test.dart: no pwsh on this host)
  • dart test -p chrome --exclude-tags local-only: 4626 pass, 1 skip (before merging main; CI's Chrome lanes cover the current head)
  • coverage-lib matrix row: not run locally; main enforces it after merge.
  • Other targeted/local-only validation: text-to-speech-models-smoke, llama-cpp-exit-teardown (quit-speaking), version-skew runs on v0.7.0-2 and v0.8.0-3, validation_coverage_test, package_skills_test.

Matrix Evidence

Matrix row Scope covered Platform / model / backend Result Evidence / notes
text-to-speech-models-smoke (published v0.9.0) All five families after merging main (#1784 landed, pin v0.9.0); no local native override macOS arm64 M4 Max, published llamadart-native v0.9.0 bundle (hook output sha256 9d9cdbc6…), CPU, 2/2 threads, isolated TMPDIR Pass (29 pass, 16 inapplicable skips) ASR transcripts exact for LFM3 default and male, Soprano, Chatterbox Turbo default and cloned, Multilingual default and cloned, Pocket default and vera; Multilingual fr gives "Bonjour, comment allez-vous aujourd'hui?". Run from the #1792 head, which contains this PR.
text-to-speech-models-smoke (pre-release native) LFM3 + Soprano synthesis, capabilities, voices, pre-synthesis refusals, seed repeat after cancel, Qwen3-ASR round trip macOS arm64 M4 Max, local llamadart-native main bee747a5, CPU, 2/2 threads; LiquidAI/LFM3-Audio-2B-GGUF Q4_K_M (+ Q8_0 mmproj), Soprano 1.1 80M converted per the guide; ASR ggml-org/Qwen3-ASR-0.6B-GGUF Q8_0 Pass (LFM3 5/5, Soprano 4/4, 1 inapplicable skip) Transcripts exact: "Good morning. The weather is sunny and warm today." for LFM3 default, LFM3 male, Soprano. Voices [female, male]; Soprano 32 kHz.
version skew: v0.8.0-3 synthesis without voice listing pinned v0.8.0-3, example CLI Pass LFM3 and Soprano speak; voices "none listed"; --voice male works; unknown voice fails with "Text-to-speech startup failed. (unknown TTS voice)".
version skew: v0.7.0-2 pre-load refusal v0.7.0-2 bundle, example CLI Pass LFM3 and Soprano throw LlamaUnsupportedException naming llama_dart_tts_init_ext and v0.8.0 before loading; Qwen3-TTS still synthesizes.
llama-cpp-exit-teardown (quit-speaking) exit() mid-synthesis macOS arm64, local llamadart-native main bee747a5 (pre-release), Metal Pass {"quit-speaking":[0,0,0]} for LFM3 and Soprano.
model-free integration masks 0xe/0x3e, legacy path, template refusal, voice listing, version messages, router/web probes VM, mirror-injected fake _TtsApi Pass test/integration/backends/llama_cpp_tts_model_types_test.dart, text_to_speech_test.dart, native_backend_test.dart.
llama-cpp-host-shutdown (Linux) N/A - Not run No Linux host; CI covers the default lanes.
chat-app-macos-quit N/A - Not run The chat app's TTS path is Qwen3-only and unchanged.
coverage-lib N/A - Not run Enforced on main after merge.
Windows / Linux / Android / iOS real-model N/A - Not run No such hosts or devices here; the Adreno guard is unchanged.

Review Notes

High-risk regression review

  • Classification: high-risk (backendRuntime, regressionPolicy, structuredOutput). structuredOutput comes only from lib/backend.dart exporting the new SPI; there is no parser, grammar or streaming change, so its row is N/A.
  • Implementation task: this PR's author session.
  • Independent blocking QA task: APPROVE at 03cdbaaa, 0 PR-caused P1/P2.
  • Exact head / current base: 03cdbaaa / merge base d51a3229. Main adds only non-overlapping commits.
  • Production-branch deletion, bypass, or miswire proof: pending audit. The integration suite fails if the opt-in mask, init_ext, start_with_voice or the pre-load probe is bypassed (the legacy-path and refusal tests assert the exact calls made).
  • Affected-family real-model/artifact evidence: LFM3-Audio and Soprano, see Matrix Evidence.
  • Explicit unavailable-family or other N/A evidence: N/A, both families were run.
  • Known PR-caused P1 regressions: 0
  • Unresolved review threads: 0

Generated from llamadart-native main eb760895 (llama.cpp cd959bae) headers,
packaged as the release workflow does, with native_header_archive.py and
ffigen in a Linux container. The same procedure reproduces the committed
v0.8.0-3 bindings byte for byte.
…type opt-in

Drive llama_dart_tts_get_info_ext/init_ext/start_with_voice with the mask of
types this core handles and the runtime builds, list predefined voices when
the runtime exports the voice API, and add Lfm3AudioTtsAdapter,
SopranoTtsAdapter, TextToSpeechRequest.voice and
TextToSpeechCapabilities.supportsVoice/voices. TextToSpeechEngine.load asks
the backend which models its runtime drives and refuses an adapter none of
whose models it drives before loading anything, naming the required
llamadart-native version.

Refs #1615.
…rano

# Conflicts:
#	website/docs/platforms/support-matrix.md
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Chat app preview removed for leehack/llamadart-chat-pr-1791.

@leehack
leehack merged commit ce8ddf6 into main Nov 8, 2026
76 of 78 checks passed
@leehack
leehack deleted the feat/tts-lfm3-soprano branch November 8, 2026 13:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant