Add offline VibeVoice ASR support - #730
Open
justinchuby wants to merge 10 commits into
Open
Conversation
Add fail-closed routing, staged ASR graphs, host processor contract, and source-backed coverage for microsoft/VibeVoice-ASR. Co-authored-by: Copilot App <[email protected]> Signed-off-by: Justin Chu <[email protected]>
Preserve final speech frames, require an unambiguous architecture, and build the source-compatible prompt and token layout. Co-authored-by: Copilot App <[email protected]> Signed-off-by: Justin Chu <[email protected]>
Preserve source prompt branches and clipping behavior while restoring the VibeVoice TTS fixture in weight alignment coverage. Co-authored-by: Copilot App <[email protected]> Signed-off-by: Justin Chu <[email protected]>
Match source clipping epsilon behavior and retain already aligned ASR parameters for direct package loading. Co-authored-by: Copilot App <[email protected]> Signed-off-by: Justin Chu <[email protected]>
Carry final-chunk state into each encoder convolution and use JSON-aware diarization extraction for source-style assistant responses. Co-authored-by: Copilot App <[email protected]> Signed-off-by: Justin Chu <[email protected]>
Treat the strict dataclass validation exception as optional so all Transformers builds remain compatible with supported Hub releases. Co-authored-by: Copilot App <[email protected]> Signed-off-by: Justin Chu <[email protected]>
Keep both normal and raw Transformers config loading compatible with older supported huggingface_hub releases. Co-authored-by: Copilot App <[email protected]> Signed-off-by: Justin Chu <[email protected]>
Emit the source tokenizer’s object-reference speech tokens and keep architecture validation compatible with older Hub releases. Co-authored-by: Copilot App <[email protected]> Signed-off-by: Justin Chu <[email protected]>
Render source-owned Qwen turns directly so compatible base tokenizers cannot inject an extra system prompt. Co-authored-by: Copilot App <[email protected]> Signed-off-by: Justin Chu <[email protected]>
Preserve terminal-frame semantics for variable-length batches and pin default ASR builds to the validated checkpoint revision. Co-authored-by: Copilot App <[email protected]> Signed-off-by: Justin Chu <[email protected]>
Performance Comparison
|
justinchuby
commented
Sep 4, 2026
Member
Author
There was a problem hiding this comment.
Move this to model docstring so it shows up in docs site
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new arch-validation test uses a huggingface_hub.hf_hub_download(local_dir=...) call that can break the same older Hub environments the PR explicitly targets for compatibility.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds first-class support for exporting and validating the original offline VibeVoice-ASR checkpoint as a staged ONNX package (dual cached waveform encoders → connectors → embedding replacement → cached Qwen2 decoder), plus host-side orchestration and ONNX Runtime GenAI advisory artifacts. This extends Mobius’ speech model coverage while keeping ASR routing fail-closed and distinct from VibeVoice TTS / streaming ASR.
Changes:
- Introduces
VibeVoiceASRForConditionalGeneration,VibeVoiceASRConfig, andVibeVoiceASRTaskto export the five-stage offline ASR pipeline. - Adds host contracts (
VibeVoiceASRProcessor,VibeVoiceASRHost) and ORT GenAI export outputs (preprocessor_config.json+ advisory compatibility metadata). - Updates Transformers builder/config loading to pin the ASR revision by default and to dispatch VibeVoice architectures explicitly (fail-closed), with expanded synthetic + arch-validation tests and documentation.
File summaries
| File | Description |
|---|---|
| tests/weight_alignment_test.py | Adds checkpoint-index census + routing coverage for the pinned VibeVoice-ASR tensor set. |
| tests/model_coverage_test.py | Documents why L4/L5 real-weight coverage is not claimed for the large BF16 checkpoint. |
| tests/build_graph/speech_test.py | Registers expected component topology for vibevoice-asr graph builds. |
| tests/arch_validation_test.py | Makes HF config loading tolerant of older Hub installs and adds coverage for that path. |
| tests/_test_configs.py | Adds a tiny VibeVoiceForASRTraining config for fast graph-level validation. |
| src/mobius/tasks/_vibevoice_asr.py | New task exporting the five offline ASR ONNX components and cache ABI. |
| src/mobius/tasks/init.py | Exposes and registers the new vibevoice-asr task. |
| src/mobius/models/vibevoice.py | Extends tokenizer encoder conv stack to support a final-chunk flag for source-equivalent padding. |
| src/mobius/models/vibevoice_asr.py | New staged offline ASR model + weight routing for both original and converted namespaces. |
| src/mobius/models/vibevoice_asr_test.py | Synthetic parity tests vs pinned Transformers VibeVoice-ASR implementation. |
| src/mobius/models/init.py | Exports VibeVoiceASRForConditionalGeneration from the models package. |
| src/mobius/integrations/vibevoice_asr.py | New host-side orchestration: normalization, chunk/cache handling, prompting, diarization parsing. |
| src/mobius/integrations/vibevoice_asr_test.py | Unit tests for processor/host behavior and deterministic orchestration contracts. |
| src/mobius/integrations/transformers/_config_resolver.py | Makes strict Hub error handling tolerant of older huggingface_hub layouts. |
| src/mobius/integrations/transformers/_config_resolver_test.py | Adds test coverage for Hub installs missing the errors module. |
| src/mobius/integrations/transformers/_builder.py | Pins microsoft/VibeVoice-ASR revision by default and enforces explicit VibeVoice architecture dispatch. |
| src/mobius/integrations/transformers/_builder_test.py | Adds fail-closed dispatch tests and validates pinned ASR raw-config build path. |
| src/mobius/integrations/onnx_genai/auto_export.py | Detects VibeVoice-ASR packages and writes processor + advisory runtime contracts. |
| src/mobius/integrations/onnx_genai/auto_export_test.py | Verifies ORT GenAI export emits processor contract + advisory compatibility for ASR. |
| src/mobius/_registry.py | Registers VibeVoiceForASRTraining as a distinct fallback architecture key and pins its test model id/revision. |
| src/mobius/_configs/vibevoice_asr.py | New config extraction for offline ASR (dual tokenizers + Qwen2 decoder) with validation. |
| src/mobius/_configs/init.py | Exposes VibeVoiceASRConfig from the configs package. |
| docs/vibevoice-asr.md | New documentation for staged offline ASR package boundaries, host contract, and evidence levels. |
| docs/model-catalog.md | Adds catalog entry linking VibeVoiceForASRTraining to the offline ASR docs. |
Review details
- Files reviewed: 24/24 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+183
to
+188
| index_path = hf_hub_download( | ||
| "microsoft/VibeVoice-ASR", | ||
| filename="model.safetensors.index.json", | ||
| revision="d0c9efdb8d614685062c04425d91e01b6f37d944", | ||
| local_dir=tmp_path, | ||
| ) |
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
VibeVoiceForASRTrainingroute formicrosoft/VibeVoice-ASR, pinned by default tod0c9efdb8d614685062c04425d91e01b6f37d944and kept distinct from VibeVoice TTS and streaming ASR.Validation
Runtime limitation
L4/L5 real-weight transcription and diarization are deliberately not claimed: the official BF16 checkpoint is approximately 8.67B parameters and no suitable GPU was available. The documentation records the CUDA/BF16 workflow required for that validation.