[leehack]

Summary

Refs #1531. Refs #1874 (0.19.4 plan, lane 4c).

Stacked on #1885 (base fix/1531-qwen3coder-seed-parity at 47a86a526, itself on #1882). Review only the commits after 47a86a526, and retarget once #1885 merges.

With ToolChoice.required, Qwen3-Coder XML grammars (Qwen3-Coder, Qwen3.5, StepFun3.5-Flash) now take what llama.cpp cd959bae9 takes (common/parsers/qwen3-coder.cpp lines 165–194):

• Text and a thought may precede the calls. The required grammar is still non-lazy, but now starts with any text that holds no call opening, the thought included, then one call (or, with parallelToolCalls, one or more). Before, it started with <tool_call>, so Qwen3.5 and StepFun3.5-Flash could not think before a required call.
• The Qwen3-Coder template's first call may leave out <tool_call>. That template has no <think>, so llama.cpp also accepts a first call that starts at a declared tool's <function=name>, and ends the content there. For a required call, the grammar, the final parse and the stream now do the same. Later calls still need <tool_call>, and templates with a thought still treat a bare <function=…> as text.
• Without a required call, a bare <function=name> stays text. This differs from llama.cpp on purpose, in the lazy grammar, the parse and the stream. llama.cpp's auto grammar triggers at that tag, and its parse ends the content there. A reply that writes the tag as text then fails in the sampler (Unexpected empty grammar stack after accepting piece), is forced into a call, or loses the rest of its answer. llamadart keeps the whole answer, as before.
• A required call can run out of budget. A thought or text may now come before the call, as in llama-server, so a reply that uses up maxTokens first ends with length and no call. The docs, the skill and the changelog say so.
• excludingRules escapes ^ as \x5E. llama.cpp's GBNF parser rejects \^, so a tool name holding ^ failed grammar init.

Root cause. buildXmlToolCallGrammar had one root for every tool choice, tool-call-scope (or tool-call-scope+), which the engine turns non-lazy for required. llama.cpp's required root is optional thought, until(call openings), calls, with <tool_call>\n optional on the first call when the template has no thought.

One intentional difference. After a thought, llama.cpp's thought rule and content rule are separate until rules whose states all accept, so a <tool_call> split across them passes: its grammar takes <tool_call> before a call, and with parallel calls off a second call. Its parser then drops what follows the first call. llamadart's exclusion rules already refuse to end text where a delimiter could start inside it, so these replies are rejected. This is recorded in test/support/qwen3_coder_required_calls.dart.

Production-readiness scope

• User-facing scope: Required tool calls on Qwen3-Coder, Qwen3.5 and StepFun3.5-Flash may think and write text first, as with llama-server. With a required call, a Qwen3-Coder reply whose first call omits <tool_call> returns that call.
• Supported platforms/paths: All. The change is in shared Dart grammar, parse and stream code (native llama.cpp and WebGPU bridge).
• Unsupported or intentionally unavailable paths:
  • The split-<tool_call> replies above: llama.cpp's grammar takes them, llamadart's does not.
  • With auto (or none), a bare <function=name> is text, not a call start (see above).
  • The call body grammar keeps its existing leniency (whitespace between tags, raw values without <). llama.cpp's body is schema-typed and newline-exact. Unchanged here.
  • StepFun3.5-Flash with thinking off: a reply with no thinking tag and no <tool_call> is reasoning in llama.cpp and content in llamadart (pre-existing, from Parse Qwen3-Coder XML replies around their tool calls as llama.cpp does (#1531) #1885's thought handling). The two fixture cases that show it are left out and noted in qwen3_coder_xml_upstream.md.
• Out of scope / follow-ups: Nemotron-3-Nano, which llama.cpp also routes to this parser, keeps llamadart's PEG-constructed path and its required grammar (thought must end with </think>). To be reported on #1874.

Completeness checklist

• Declared scope is fully implemented, or explicitly reduced above.
• Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable. (No new unsupported path.)
• Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant. (Both changelogs, tool-calling.md, template-engine-internals.md, skills/llamadart-tool-calling/SKILL.md. No public API change.)
• Regression coverage covers the original issue plus key negative/version-skew paths where relevant.
• Public capability changes update the validation coverage registry with a pack or a reasoned, issue-linked exemption; inventory checks pass (or N/A with scope). N/A: grammar parity fix, no capability change.
• Security/privacy review completed: no secrets, bearer tokens, signed URLs, or raw secret-bearing paths leak through logs, cache keys, metadata, errors, or snapshots.
• Follow-up work that is useful but not required for this PR is tracked in GitHub Issues, or explicitly marked "None" above.

PR type guidance

• Bugfix PR: root cause and fix above.
  • Grammar acceptance (test/support/qwen3_coder_required_calls.dart): one list of accepted and rejected replies per template config, built from llama.cpp's own required grammars. It runs through the Dart GbnfRecognizer in test/integration/core/template/qwen3_coder_required_grammar_test.dart (CI, VM) and through llama.cpp's test-gbnf-validator in specialized_tool_grammar_validation_e2e_test.dart (local-only). The configs are Qwen3-Coder, Qwen3.5-0.8B (thinking on and off) and StepFun3.5-Flash (on and off), each with parallel off and on. The cases cover: a call; text then a call; a thought then text then a call; </think> twice; <function=unknown> in text; parallel calls; a bare first call (Qwen3-Coder only); and as rejections, no call, text after the calls, an unknown tool, a bare second call, a second call without parallel, and a bare call where llama.cpp takes it as text.
  • Upstream parse fixture. qwen3_coder_xml_upstream.json gains 6 cases per config, recorded with the same cd959bae9 probe: bare-call, text-then-bare-call, bare-call-then-call, call-then-bare-call, unknown-bare-function-in-text, longer-bare-function-in-text. The probe first reproduced all 180 existing records (0 mismatches). The integration test checks each case's parse, partial parse and every stream chunking.
  • Real model. llama_cpp_chat_template_backend_e2e_test.dart (local-only) loads Qwen3.5-0.8B-Q4_K_M with its own template. It makes a required call with thinking on and off. The grammar must compile in the native sampler and give tool_calls, and with thinking on the reply must have reasoning first.
  • ToolCallGrammarUtils.excludingRules now handles several delimiters. A brute-force test checks it on all strings up to length 5 for four delimiter sets. Its output for one delimiter is byte-identical to the old _excludingRules (checked on 11 delimiters).

Test Plan

• dart run tool/prepare_workspace.dart
• dart format --output=none --set-exit-if-changed .
• dart analyze, plus dart run tool/testing/check_platform_boundaries.dart
• dart test -p vm -j 1 --exclude-tags local-only (isolated TMPDIR, at b75aa2e91): 12,782 passed, 11 skipped, and 6 failed. All 6 are windows_registry_qualification_test.dart, which needs pwsh and this host has none
• dart test -p chrome --exclude-tags local-only (at b75aa2e91): 4,667 passed, 1 skipped
• coverage-lib: run by the dispatched CI below (ci.yml collects coverage on manual runs)
• Local-only, at b75aa2e91: specialized_tool_grammar_validation_e2e_test.dart with the v0.9.0 (cd959bae9) test-gbnf-validator: 81 passed. llama_cpp_chat_template_backend_e2e_test.dart on Qwen3.5-0.8B-Q4_K_M, CPU: 4 passed
• ./tool/testing/run_template_parity_suites.sh (at b75aa2e91): detection 74, diagnostic 103, unit template 1,875, upstream parser 10, GBNF acceptance 81, all passed

Matrix Evidence

Matrix row	Scope covered	Platform / model / backend	Result	Evidence / notes
root-vm	native-safe unit/integration	macOS arm64 VM	PASS	12,782 passed, 11 skipped, 6 pwsh-only failures
root-chrome	browser-compatible tests	Chrome	PASS	4,667 passed, 1 skipped
template-parity	detection, render and parse parity; GBNF acceptance	macOS arm64	PASS	detection 74, diagnostic 103, unit template 1,875, upstream parser 10, GBNF acceptance 81
structured-output-adversarial	compiled grammar acceptance and rejection; required and auto tool choice; thinking on and off; parallel on and off; parse, partial and every-chunking stream against llama.cpp	llama.cpp cd959bae9 grammars and common_chat_session oracle; Qwen3.5-0.8B-Q4_K_M	PASS	Differential below
high-risk-exact-head-independent-qa	independent blocking review of the exact head	—	PENDING	Required before mark-ready
coverage-lib	lib/ line coverage	Linux CI	PASS	Dispatched ci.yml run
Other rows	—	—	N/A	No release, LiteRT-LM or backend runtime change

Grammar differential. The 10 configs' llama.cpp required grammars came from the cd959bae9 probe. Each reply ran through test-gbnf-validator with the generation prompt prefilled, as llama.cpp's sampler does (common/sampling.cpp). llamadart's grammar ran on the same reply.

Measure	Count
Replies checked	212
llama.cpp result = expected	203
llama.cpp quirk (split <tool_call> after a thought, accepted)	9
llamadart result = expected	212

Failing first. The new and updated tests were run against base 47a86a526 lib. 19 fail:

• all 10 required-grammar configs (text before the call is rejected)
• the lazy bare-call trigger test and the grammar root test
• the Qwen3.5 forced-open required_grammar_start
• the 6 Qwen3-Coder bare-call, text-then-bare-call and bare-call-then-call fixture cases

The real-model test fails under mutant M1 (the old root): the reply has no thought before the call.

Parse sweep against llama.cpp. The corpus is all fixture outputs plus 8 bare-tag prose replies, 32 in all. It runs on 20 configs: each template, thinking setting and parallel setting, with auto and required. That is 640 records, compared with the cd959bae9 common_chat_session finished parse.

Revision	= llama.cpp
base 47a86a526	526
55c69df04	556
head b75aa2e91	538

Every changed record is intended:

• Head vs base: only 18 Qwen3-Coder required records change.
  • 12 now match llama.cpp: bare first calls.
  • 6 still differ. In these, required has no valid call, so llama.cpp fails the reply and llamadart returns content (pre-existing, noted above). The required grammar does not let the model write these replies.
• Head vs 55c69df04: only the 18 Qwen3-Coder auto records holding a bare <function=get_weather> change. They go back to base behaviour: the tag and what follows are content.

Real-model regressions. Both use Qwen3.5-0.8B-Q4_K_M on CPU, in llama_cpp_chat_template_backend_e2e_test.dart:

• Required call, with the model's own template, thinking on and off. It fails under M1, the old root: the reply has no thought before the call.
• ToolChoice.auto with the Qwen3-Coder template and the prompt "explain what the literal text <function=get_weather> means". The reply must have no call, no error, and the answer after the tag.
  • At 55c69df04 it fails: the content is only <think>\n.
  • With the <function=name> trigger restored, llama.cpp's sampler throws Unexpected empty grammar stack after accepting piece.

Mutations: 18 of 18 killed at b75aa2e91.

Mutant	Where
M1 required root without leading text	handler
M2 first scope start mandatory	grammar builder
M3 bare openings allowed in leading text	grammar builder
M4 every exclusion state may end	excludingRules
M5 no inherited delimiter ends	excludingRules
M6 bare first call also with a thought	handler
M7b <function=name> lazy trigger in auto (llama.cpp's trigger)	handler
M8 parse ignores the bare call	handler parse
M9 bare call wins over an earlier <tool_call>	handler parse
M10 stream ignores bare openings	stream parser
M11 copies drop the mark	withThoughtMarksOf
M12 parseRendered drops the mark	engine
M14 _parse drops the flag	engine
M15 bare openings only without leading text	grammar builder
M17 ^ escaped as \^ (also fails the validator: unknown escape at \^])	excludingRules
M18 leading text for Nemotron-3-Nano too (audit A15)	handler
M19 bare first call without a required call (re-audit)	handler
M20 bare opening without its > (audit A12, <function=get_weathers>)	format

M13 is the mark copied by the stream's forced-open copy. It survived at 03f40f9ce because that copy can never run, so the copy was removed. M16 (the lazy grammar takes a bare first call) is now the same mutant as M19.

Audit follow-ups.

• 55c69df04 fixed the three findings from the audit at 95d4986c9: the auto trigger, the required budget docs and the ^ escape. It also pinned A15.
• b75aa2e91 follows the re-audit at 55c69df04. It takes a bare first call only for a required call (maintainer decision), and pins A12.

Review Notes

• Independent review status: audit at 95d4986c9 PASS with three findings, fixed in 55c69df04. Re-audit at 55c69df04 PASS with one change, made in b75aa2e91. A re-audit of b75aa2e91 is pending.
• CI status / head SHA: b75aa2e91, dispatched ci.yml run 42796454762 (success, coverage gate included).

High-risk regression review

• Classification: high-risk (backendRuntime, structuredOutput)
• Implementation task: lane 4c of #1874
• Independent blocking QA task: audit at 95d4986c9 and re-audit at 55c69df04: PASS, with findings fixed in 55c69df04 and b75aa2e91. Re-audit of b75aa2e91 pending
• Exact head / current base: b75aa2e91 / fix/1531-qwen3coder-seed-parity 47a86a526
• Production-branch deletion, bypass, or miswire proof: the 18 killed mutants above, covering the grammar builder, exclusion rules, handler, parse, engine and stream call sites
• Affected-family real-model/artifact evidence: Qwen3.5-0.8B-Q4_K_M real-model required calls (thinking on and off, native CPU). The cd959bae9 grammars and parses cover the vendored Qwen3-Coder, Qwen3.5 and StepFun3.5-Flash templates.
• Explicit unavailable-family or other N/A evidence: Qwen3-Coder and StepFun3.5-Flash weights were not used. Their evidence is the primary llama.cpp grammars and parses on their vendored templates.
• Known PR-caused P1 regressions: 0
• Unresolved review threads: 0

[leehack]

Landed on main via batch integration PR #1895 (rebase merge, main ba4e0e1b; tree identical to the audited integration head 5ffd1224).