Skip to content

fix(asr): force Qwen3-ASR language with the official prefix, not garbled task tokens - #1

Merged
msandroid merged 1 commit into
translateblue/parakeet-ctc-ja-vocabfrom
claude/qwen3-asr-language-forcing
Sep 24, 2026
Merged

msandroid merged 1 commit into
translateblue/parakeet-ctc-ja-vocabfrom
claude/qwen3-asr-language-forcing

Conversation

@msandroid

Copy link
Copy Markdown
Owner

Why is this change needed?

When Qwen3AsrManager.transcribe(language:) got a language, it put taskTokens[lang] into the system turn. The comment says those are "Transcribe the audio to {Language} text.", but decoded with the Qwen3-ASR vocab (FluidInference vocab.json is byte-identical to Qwen/Qwen3-ASR-0.6B) every one of the 30 rows is gibberish:

language what the model actually received as system text
English agnıs the Oper to English text.
Korean agnıs the Oper to essentially text.
Cantonese agnıs the Oper to(signal.sender text.
Japanese agnıs the Oper toPoints text.

The official qwen_asr package doesn't use the system turn for this at all: the system turn is context (biasing text), and a language is forced by appending language {Name}<asr_text> right after <|im_start|>assistant\n (Qwen3ASRModel._build_text_prompt, force_language). TranslateBlue passes explicit hints for Korean and Cantonese, where Qwen3 is the tier-1 ASR, so every such session was injecting this noise.

What changed

  • Qwen3AsrPrompt (new): builds the official template with an empty system turn and, when a language is given, appends the forced prefix. The 30 token rows were generated by tokenizing the official prompt with the Qwen/Qwen3-ASR-0.6B tokenizer (fix_mistral_regex=True, as qwen_asr loads it); each row decodes back to exactly language {Name}<asr_text> (checked for all 30).
  • Qwen3AsrManager uses it; taskTokens / buildPromptTokens are removed. With language == nil the prompt tokens are unchanged, so auto-detect behaves as before. decodeTokens already handles output with no <asr_text>, which is what forced decoding produces.
  • Qwen3AsrPromptTests: pins the auto/forced token layout against the official tokenization and checks the table's invariants.
  • Doc comment and Documentation/ASR/Qwen3-ASR.md wording.

Branching

Branched from cf46be1, the commit TranslateBlue currently pins, so bumping the submodule to this fix doesn't pull in the later Canary / SenseVoice commits. Nothing under ASR/Qwen3 changed between cf46be1 and the branch head, so this merges cleanly into translateblue/parakeet-ctc-ja-vocab.

Not verified

  • Not compiled. The editing environment is Linux with no Swift toolchain, and FluidAudio needs CoreML. Before merging, run swift build and swift test --filter "Qwen3AsrPromptTests|Qwen3AsrConfigTests" on a Mac.
  • Accuracy not measured. Suggested A/B on a Mac: swift run -c release fluidaudiocli qwen3-transcribe <ko.wav> --language ko on FLEURS ko/yue before and after this change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UDraWTxHPu3AegNzaFrowi


Generated by Claude Code

…led task tokens

Qwen3AsrManager put `taskTokens[lang]` in the system turn when a language was
given. Decoded with the Qwen3-ASR vocab (identical to the bundled vocab.json)
every entry was gibberish ("agnıs the Oper to essentially text." for Korean,
"...toPoints text." for Japanese — all 30 languages), and the official
qwen_asr package does not use the system turn for this anyway: it is context,
and a language is forced by appending `language {Name}<asr_text>` after the
assistant header.

- Qwen3AsrPrompt builds the official template (empty system turn) and appends
  the forced-language prefix; the 30 token rows were generated with the
  Qwen/Qwen3-ASR-0.6B tokenizer (fix_mistral_regex=True) and each decodes back
  to exactly `language {Name}<asr_text>`
- language == nil produces the same tokens as before (auto-detect unchanged)
- Qwen3AsrPromptTests pin the auto/forced token layout and table invariants

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01UDraWTxHPu3AegNzaFrowi

Copy link
Copy Markdown
Owner Author

swift-format failure: runner infrastructure, not this change.

Evidence it never ran against the code:

  • The job ended 14 s after starting, with runner_id: 0 and no runner name, so no macos-15 runner was ever assigned. The job log isn't available either (HTTP 404).
  • Every macos-15 job in this fork has failed the same way since August. All four Build and Test jobs on main (runs 33288082510, 33738629289, 34611866398, 34764645049) failed within 3–7 s. No fix exists on the base branch, and nothing in this PR can change runner availability.

Status:

  • I've used the one re-run (attempt 2, queued).
  • Formatting of this diff is still unverified. The editing environment has no Swift toolchain, which is also why this PR is unbuilt. Before merging, please run on a Mac:
swift format lint --recursive --configuration .swift-format Sources/FluidAudio/ASR/Qwen3 Tests/FluidAudioTests/ASR/Qwen3
swift build
swift test --filter "Qwen3AsrPromptTests|Qwen3AsrConfigTests"

Generated by Claude Code

@msandroid
msandroid marked this pull request as ready for review September 24, 2026 15:28
@msandroid
msandroid merged commit 408ebf8 into translateblue/parakeet-ctc-ja-vocab Sep 24, 2026
0 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants