Add VibeVoice streaming ASR export - #723
Open
justinchuby wants to merge 2 commits into
Open
Conversation
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
justinchuby
force-pushed
the
justinchuby-add-vibevoice-streaming-asr
branch
from
September 3, 2026 22:56
9f1f808 to
87138b0
Compare
Implement an architecture-discriminated, staged ONNX export for the pinned VibeVoice ASR checkpoint, with explicit audio convolution and decoder cache contracts. Preserve the existing TTS path while recording unsupported ORT GenAI orchestration boundaries and validating source parity, processor behavior, and complete checkpoint routing. Co-authored-by: Copilot App <[email protected]> Signed-off-by: Justin Chu <[email protected]>
Apply the repository import and expression formatting rules after restacking the streaming ASR export on the finalized VibeVoice TTS base. Co-authored-by: Copilot App <[email protected]> Signed-off-by: Justin Chu <[email protected]>
justinchuby
force-pushed
the
justinchuby-add-vibevoice-streaming-asr
branch
from
September 3, 2026 22:57
87138b0 to
e6c4950
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack dependency
This PR is intentionally stacked on #714 (
justinchuby-add-vibevoice-tts), at base commit346988887d84a1964a3d7d286e4ff200f38d302e. It must merge only after #714; it does not enable auto-merge.Implementation
VibeVoiceForASRStreamingTrainingroute formicrosoft/VibeVoice-ASR-Streaming-7B, without changing the existing VibeVoice TTS route.audio_encoder(acoustic and semantic causal tokenizers plus connectors),embedding(speech embedding replacement), anddecoder(Qwen2 with explicit KV caches).<speech_start>,<speech_end>, forced<|text_chunk_end|>), chunk loop/sampling, hotword prompt assembly, and speaker JSON parsing remain host-owned.mobius.attention.requires_arbitrary_maskcontract, preventing incorrect GQA fusion for left-padded ASR batches.acoustic_tokenizer.decoder.*training-VAE tensors excluded from inference.Provenance and runtime boundary
microsoft/VibeVoice-ASR-Streaming-7B@60d858b518b4e19d404af3737f848fc185b30177.microsoft/VibeVoice@505653d3873b065a488aea551c6ee3dc51d3062fwithtransformers==4.51.3; ASR-critical blobs were checked unchanged through upstream1541f590c7099820f10ea012f48d2399282df69f.onnxruntime-ep-mlx 0.29.5/ ORT 1.29 also cannot execute the complete VibeVoice decoder due toCumSum cannot infer output shapes; this blocks local MLX real-runtime evidence.Validation
9951 passed, 66 skippedfull non-integration suite.lintrunner f --output oneline --all-filesand diff checks.