Skip to content

0.19.4 batch integration: LiteRT-LM Web tool calling and llama.cpp parity fixes - #1895

Merged
leehack merged 15 commits into
mainfrom
integration/0.19.4
Nov 10, 2026
Merged

leehack merged 15 commits into
mainfrom
integration/0.19.4

Conversation

@leehack

@leehack leehack commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Batch integration PR for the 0.19.4 plan (#1874), built per doc/high_risk_pre_merge_readiness.md "Batch integration PR": a new branch from main at 77b01da1f6db7cde751b618700574b63f3eb74a9, one commit per constituent (its own delta at the pinned head, applied in stack order), conflicts resolved once inside the applying commit.
  • Land with a rebase merge, never squash. Close constituents only after the merge, with a link here.
  • Ready: integration audit PASS at 5ffd1224 (fresh-agent-auditor-1895-integration), 0 PR-caused P1, 0 unresolved threads here and on every constituent; evaluator exit 2 (llamadart-evidence/pr-1895/).

Closes #1868
Closes #1790
Closes #919
Closes #1865
Closes #1887
Closes #1892

#1863 stays open (#1876 only documents it); #1861 and #1531 are referenced, not closed, by their constituents.

Constituents

Each commit's diff equals its constituent's delta: git diff <base> <pinned head>, where the base is the merge base with main for a PR into main, and the parent's pinned head for a stacked PR (#1886 merged main 77b01da1f in, so its base is current main). Checked with git diff -U0 (hunk headers stripped) per commit: identical except the listed resolutions.

Commit PR Risk (surfaces) Pinned head Delta base Gate
9a433443a #1875 high (backendRuntime) 03d50e48e8f9c40bf1bb56f0116513049a8a26d7 f78cafa97 Audit accepted at 03d50e48e / base f78cafa97 (fresh-agent-auditor-1875, 0 P1, 0 threads); CI green at head
dcfdb118c #1888 high (backendRuntime) 1682ba906fb6b6546f90a77b49e2adbe8e44cdc5 #1875 03d50e48e Audit accepted at 1682ba906 / base 03d50e48e (fresh-agent-auditor-1888, 0, 0); stacked, so no PR CI
d0553545a #1876 standard a05d460caea648d2423e99b7d1d3ca9aca731335 f78cafa97 CI run 42733309914 success at head
d6ffe46d4 #1877 standard ee7a66cbf823e326b9bafeda3974cb14c46d1e6d f78cafa97 CI run 42735121670 success at head
472ae0cf4 #1878 high (regressionPolicy, structuredOutput) 691a2a373d89dc0f0d7ff65d3799bb4cb65455a9 f78cafa97 Audit accepted at 691a2a373 / base f78cafa97 (fresh-agent-auditor-1878, 0, 0); CI green at head
5f6948b19 #1879 high (backendRuntime) 2ebe8141ca024e17a0a4f799151d9a57228ff10e f78cafa97 Audit accepted at 2ebe8141c / base f78cafa97 (fresh-agent-auditor-1879, 0, 0); CI green at head
c29d15af9 #1880 high (backendRuntime, regressionPolicy) 7d4630ffd5fbf32a2e4b2becdbb6c3d0b63067ef f78cafa97 Audit accepted at 7d4630ffd / base f78cafa97 (fresh-agent-auditor-1880, 0, 0); CI green at head
b1943832a #1891 high (backendRuntime, regressionPolicy) de450fcc5720ae40914b6b71f4c4d644fd723449 #1880 7d4630ffd Audit accepted at de450fcc5 / base 7d4630ffd (fresh-agent-auditor-1891b, 0, 0); stacked, so no PR CI
7a4d03bfd #1882 high (backendRuntime, structuredOutput) a6ea0316b6913b247fe4a744d0ebbfa152088a2d f78cafa97 Audit accepted at a6ea0316b / base f78cafa97 (fresh-agent-auditor-1882, 0, 0); CI green at head
78cc06faa #1885 high (backendRuntime, structuredOutput) 47a86a526fe3bd83844a9c77243dc3221b18aa61 #1882 a6ea0316b Audit accepted at 47a86a526 / base a6ea0316b (fresh-agent-auditor-1885, 0, 0); stacked, so no PR CI
18194b768 #1890 high (backendRuntime, structuredOutput) b75aa2e91e71d39b21ccfda370c6a8df730fb378 #1885 47a86a526 Re-audit accepted at b75aa2e91 / base 47a86a526 (fresh-agent-auditor-1890-reaudit-b75aa2e, 0, 0); stacked, so no PR CI
eb7d08908 #1894 high (backendRuntime, structuredOutput) 302757be6c3adb22e23d8731bb6df9fe072b2517 #1890 b75aa2e91 Accepted: delta re-audit at 302757be6 / base b75aa2e91, 0 P1, 0 threads, evaluator exit 2 (evidence pr-1894/). Stacked, so dispatched CI only (run 42802975044, green)
03e35bcea #1886 high (structuredOutput) 67d16515b74f314d55aff2160c614a344f5d32da 77b01da1f (main merged in) Re-audit accepted at 67d16515b / base 77b01da1f (fresh-agent-auditor-1886-reaudit-2, 0, 0); CI green at head
17409e928 #1889 high (releaseAutomation) e8e72902536fa15776961a9b6ac5f40d000c8013 77b01da1f Audit accepted at e8e729025 / base 77b01da1f (fresh-agent-auditor-1889, 0, 0); CI green at head
5ffd12244 #1893 high (regressionPolicy) 34f8911daaf2e8ff1ed29a82962d74cbd06b27c9 77b01da1f Audit accepted at 34f8911da / base 77b01da1f (fresh-agent-auditor-1893, 0, 0); CI green at head

Risk is classify_high_risk_changes.dart on each delta. Audit records are the constituents' v2 evidence documents (independent_audit). Stacked PRs target their parent's branch, so ci.yml never ran on them; their only check is the read-only advisory, which fails by design for a non-default base ("Diagnostic is restricted to PRs targeting the repository default branch"). CI on this PR is their first CI run.

Unresolved review threads per constituent (live count at merge time): to be recorded before merge.

Conflict resolutions

Every other hunk is the constituent's own, unchanged.

Commit File Hunk Sides Resolution
c29d15af9 (#1880) website/docs/guides/backend-selection.md "LiteRT-LM Web rejects ..." paragraph, last five lines #1876 replaced "Raw generate sends one user text." with "Raw generate wraps the prompt in one user turn of the bundle's template ([#1863])"; #1880 replaced "Automatic tool loops remain unavailable. The remaining #1152 scope is tracked in [the runtime gap]" with "Automatic tool loops remain unavailable ([#919])." Kept both: #1876's generate sentence, then #1880's tool-loop sentence.
b1943832a (#1891) website/docs/guides/backend-selection.md same paragraph, last five lines #1876's generate sentence (as above); #1891 replaced "Automatic tool loops remain unavailable ([#919])." with "The default constructors report no token-limit truncation and refuse automatic tool loops." Kept both: #1876's generate sentence, then #1891's sentence.
9a433443a (#1875), d0553545a (#1876), 5f6948b19 (#1879), c29d15af9 (#1880), 7a4d03bfd (#1882), 03e35bcea (#1886) CHANGELOG.md and website/docs/changelog/recent-releases.md (one hunk each) ## Unreleased Constituent added its bullet plus a blank line to an empty ## Unreleased; main cut 0.19.3 below it and earlier commits here already added bullets. Bullet appended to the ## Unreleased list without the extra blank line. The union driver had put #1875's bullet under ## 0.19.3; every bullet here was placed by hand under ## Unreleased.
b1943832a (#1891), 78cc06faa (#1885), 18194b768 (#1890), eb7d08908 (#1894) both changelogs (one hunk each) ## Unreleased Constituent bases predate the 0.19.3 cut. Bullet appended under ## Unreleased; text identical to the constituent's line.

Result: ## Unreleased holds 10 bullets in both files, one per changelog-bearing constituent, in commit order, with no duplicates; released sections equal main (check_released_changelog_sections.dart passes).

Nothing was sent back to a constituent.

Production-readiness scope

  • User-facing scope: the union of the constituents: LiteRT-LM Web manual tool calling (#1880) and token-limit truncation with automatic tool loops (#1891); native LiteRT-LM send-failure template note and argument-error names (#1875, #1888, #1879); llama.cpp-parity parsing for thoughts (#1882, #1894), Qwen3-Coder XML (#1885, #1890) and Seed-OSS (#1886); docs (#1876); tests and CI tooling (#1877, #1878, #1889, #1893).
  • Supported platforms/paths: as stated in each constituent PR.
  • Unsupported or intentionally unavailable paths: as stated in each constituent PR.
  • Out of scope / follow-ups: #1863, #1861, #1531 as tracked by the constituents.

Completeness checklist

  • Declared scope is fully implemented, or explicitly reduced above. (Per constituent.)
  • Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable. (Per constituent.)
  • Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant. (Per constituent; changelog merged as above.)
  • Regression coverage covers the original issue plus key negative/version-skew paths where relevant. (Per constituent.)
  • Public capability changes update the validation coverage registry with a pack or a reasoned, issue-linked exemption; inventory checks pass (or N/A with scope). (Add LiteRT-LM Web manual tool calling on @litert-lm/core 0.19.0 #1880, Report LiteRT-LM Web token-limit truncation and run automatic tool loops there #1891.)
  • Security/privacy review completed: no secrets, bearer tokens, signed URLs, or raw secret-bearing paths leak through logs, cache keys, metadata, errors, or snapshots. (Per constituent audit.)
  • Follow-up work that is useful but not required for this PR is tracked in GitHub Issues, or explicitly marked "None" above.

PR type guidance

  • Feature / Bugfix / Docs PR: N/A here; each constituent PR carries its own root cause, tests and docs.

Test Plan

Local runs on the integration head, macOS arm64, Flutter 3.47.1 / Dart 3.13.1, isolated TMPDIR:

  • dart run tool/prepare_workspace.dart: tracked tree clean afterwards
  • dart format --output=none --set-exit-if-changed .: 0 changed
  • dart analyze: no issues
  • dart run tool/testing/check_platform_boundaries.dart: OK
  • check_released_changelog_sections.dart (pull_request event, base main): released sections match the base
  • dart run tool/testing/verify_release_docs_versions.dart: verified (llamadart and companions 0.19.3, native v0.9.0)
  • dart test -p vm -j 1 --exclude-tags local-only: 12910 passed, 11 skipped, 6 failed. All 6 are test/unit/tooling/windows_registry_qualification_test.dart cases failing with ProcessException: No such file or directory for pwsh, which this Mac lacks; no constituent touches that test or its scripts, and CI's VM and Windows tooling shards pass.
  • dart test -p chrome --exclude-tags local-only: 4739 passed, 1 skipped
  • coverage-lib: 93.68% (33572/35837) against the 70% floor (same 6 pwsh failures in the coverage run)
  • Chat app: flutter test test test_vm 694 passed, 8 skipped; flutter test --platform chrome on the four Web files 136 passed, 1 skipped
  • tool/testing/run_template_parity_suites.sh: detection 74, diagnostics 103, unit template 1887, upstream llama.cpp chat suites 10, compiled grammar acceptance 81; all passed
  • Clean combination: each commit's git diff -U0 equals its constituent's delta except the changelog hunks listed above; file sets identical

Matrix Evidence

Matrix row Scope covered Platform / model / backend Result Evidence / notes
high-risk-exact-head-independent-qa integration audit of this exact head vs current main — PASS Tree = main + 15 pinned deltas + listed resolutions; parity suites, GBNF e2e, 37/38 mutants, real-model smokes (LiteRT-LM Web Qwen3-0.6B CPU / Gemma 4 WebGPU, native LiteRT-LM Qwen3-0.6B, Qwen3.5-0.8B GGUF) all pass.
static-format-analyze format, analyzer, platform boundaries macOS arm64 PASS local, integration head
root-vm VM unit/integration macOS arm64 PASS (6 host-only pwsh failures) local and CI, integration head
root-chrome browser unit/integration macOS Chrome PASS local and CI, integration head
structured-output-adversarial template/parser/grammar parity vendored llama.cpp templates, compiled GBNF PASS tool/testing/run_template_parity_suites.sh on the integration head; real-model and affected-family evidence per constituent (#1882, #1885, #1890, #1894, #1886) and for the integration audit
coverage-lib lib/ line coverage macOS arm64 PASS (93.68%)
release-metadata-verification N/A — N/A Not a release-prep PR.
litert-lm-v1-public-direct / -service / -speech N/A here — N/A Native LiteRT-LM evidence is per constituent (#1875, #1888, #1879); integration audit decides on re-runs.
litert-lm-json-schema N/A — N/A No native schema-constraint change.

Review Notes

High-risk regression review

  • Classification: high-risk (backendRuntime, regressionPolicy, releaseAutomation, structuredOutput)
  • Implementation task: the constituent PRs; this PR only combines them.
  • Independent blocking QA task: integration audit, pending.
  • Exact head / current base: 5ffd12244bb60804d9c42c605940ccc65d28f31e / 77b01da1f6db7cde751b618700574b63f3eb74a9
  • Production-branch deletion, bypass, or miswire proof: per constituent evidence; integration evidence pending.
  • Affected-family real-model/artifact evidence: per constituent; integration audit runs the relevant smokes on the combined tree.
  • Explicit unavailable-family or other N/A evidence: per constituent.
  • Known PR-caused P1 regressions: 0 known (integration audit pending)
  • Unresolved review threads: 0 on this PR; per-constituent live counts recorded at merge time.

… no thought, as llama.cpp does (#1882)

Delta of #1882: pinned head a6ea031 against base f78cafa.
@github-actions

Copy link
Copy Markdown
Contributor

Chat app preview removed for leehack/llamadart-chat-pr-1895.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment