Skip to content

Explain which chat template LiteRT-LM bundles render (#1836) - #1842

Closed
leehack wants to merge 1 commit into
mainfrom
docs/1836-litert-chat-template-note
Closed

leehack wants to merge 1 commit into
mainfrom
docs/1836-litert-chat-template-note

Conversation

@leehack

@leehack leehack commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Docs-only clarification for #1836: on native .litertlm, getMetadata() and engine.chatTemplate(...) show the file-name family template that drives Dart-side format detection and parsing. The runtime renders that template only for text-only Qwen3 requests. Qwen3.5, Gemma and every request with media render the bundle's embedded template, so the native prompt can differ.
  • Updates website/docs/getting-started/model-families.md (the .litertlm bullet and the "To see what a model will receive" note) and skills/llamadart-tool-calling/SKILL.md.
  • Refs LiteRT-LM .litertlm: separate the reported/rendered bundle chat template from the Dart-side format and capability template #1836. A code fix would change format detection and capability results. The issue is now the post-release design issue for that fix; see the findings comment there.

Production-readiness scope

  • User-facing scope: the docs say accurately what chatTemplate()/metadata show for native LiteRT-LM bundles.
  • Supported platforms/paths: docs-only (website and the published agent skill).
  • Unsupported or intentionally unavailable paths: N/A. Runtime behavior is unchanged.
  • Out of scope / follow-ups: report the rendering template without changing parsing: #1836.

Completeness checklist

  • Declared scope is fully implemented, or explicitly reduced above.
  • Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable. (This PR documents the gap.)
  • Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant. (No changelog bullet: docs clarification only.)
  • Regression coverage covers the original issue plus key negative/version-skew paths where relevant. N/A: no behavior change.
  • Public capability changes update the validation coverage registry with a pack or a reasoned, issue-linked exemption; inventory checks pass (or N/A with scope). N/A: no capability change.
  • Security/privacy review completed: no secrets, bearer tokens, signed URLs, or raw secret-bearing paths leak through logs, cache keys, metadata, errors, or snapshots.
  • Follow-up work that is useful but not required for this PR is tracked in GitHub Issues, or explicitly marked "None" above.

PR type guidance

  • Feature PR: N/A
  • Bugfix PR: N/A
  • Docs-only PR: no runtime behavior change. I checked the wording against litert_lm_service.dart (useTextTemplate/promptTemplate) and the ModelParams.chatTemplate Dartdoc, and verified it on real bundles (Review Notes). The docs build and link check pass.

Test Plan

  • dart run tool/prepare_workspace.dart
  • dart format --output=none --set-exit-if-changed . (0 changed)
  • dart analyze: N/A, Markdown-only diff (CI runs it)
  • dart test -p vm -j 1 --exclude-tags local-only: N/A, ran the Markdown-selected suites instead (below)
  • dart test -p chrome --exclude-tags local-only: N/A, docs-only
  • coverage-lib matrix row when lib/ behavior changes: N/A, no lib/ change
  • Other targeted/local-only validation: dart test -p vm -j 1 $(cat tool/ci/markdown_test_suites.txt) (728 passed, including package_skills_test.dart); ./tool/docs/build_site.sh --port 18736 (exit 0, link check passed)

Matrix Evidence

Matrix row Scope covered Platform / model / backend Result Evidence / notes
docs-site website build, docs metadata, generated docs routes macOS arm64, Flutter 3.47.1 PASS ./tool/docs/build_site.sh --port 18736 exit 0 (the default port 8080 was taken locally)

Review Notes

  • Independent review status: N/A, standard-risk docs-only change.
  • Real-bundle facts behind the wording (macOS arm64, native LiteRT-LM CPU):
    • Qwen3.5-0.8B_int8.litertlm: metadata reports the Qwen3 template (hermes). chatTemplate([user 'hi'], enableThinking: true) ends in <|im_start|>assistant\n. The embedded template, which the runtime renders, always ends in <think>\n\n</think>\n\n.
    • gemma-4-E2B-it.litertlm: the embedded template (11995 chars) differs from llamadart's gemma4 template (16430 chars).
  • CI status / head SHA: green at 278b8dba7 (32 pass, 9 skipped, 0 failed).

High-risk regression review

  • Classification: standard
  • Implementation task: N/A
  • Independent blocking QA task: N/A
  • Exact head / current base: N/A
  • Production-branch deletion, bypass, or miswire proof: N/A
  • Affected-family real-model/artifact evidence: N/A
  • Explicit unavailable-family or other N/A evidence: N/A
  • Known PR-caused P1 regressions: 0
  • Unresolved review threads: 0

For native .litertlm, getMetadata() and chatTemplate() show the
file-name family template that drives Dart-side parsing; the runtime
renders it only for text-only Qwen3 requests and uses the bundle's
embedded template otherwise.

Refs #1836
@leehack

leehack commented Nov 9, 2026

Copy link
Copy Markdown
Owner Author

Landed via batch integration PR #1843 (rebase merge), one commit per constituent.

@leehack leehack closed this Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Chat app preview removed for leehack/llamadart-chat-pr-1842.

leehack added a commit that referenced this pull request Nov 9, 2026
* Raise the Apple companions' Flutter floor to 3.38.6 (#1834) (#1841)

Squash of #1841 at 345ada9 against merge-base ec4bf4f.

* Explain which chat template LiteRT-LM bundles render (#1836) (#1842)

Squash of #1842 at 278b8db against merge-base ec4bf4f.

* Record LiteRT-LM thinking as N/A for bundle templates that cannot think (#1833)

Squash of #1833 at eda933e against merge-base ec4bf4f.

* Pin Windows pending-delete removal for remove and prune (#1837) (#1839)

Squash of #1839 at 403835b against merge-base b78008c.

* Pin Chrome supervisor exit precedence and post-done wait (#1837 a-d) (#1840)

Squash of #1840 at 73c66a0 against merge-base ec4bf4f.

* Refuse LiteRT-LM GPU on Linux when every Vulkan device is software (#1831)

Squash of #1831 at e51e76c against merge-base ec4bf4f.

* Require the in-process LiteRT marker for Linux and Windows LiteRT-LM embeddings (#1830)

Squash of #1830 at 0296fd5 against merge-base b78008c.

Conflict resolutions:
- website/docs/platforms/known-platform-issues.md, "## Linux and Windows", custom-runtime bullet tail and the Linux x64 GPU bullet (vs #1831): #1830's embeddings-marker paragraph replaces the legacy `v0.18.x` refusal paragraph, followed by #1831's rewritten Linux x64 software-Vulkan GPU bullet (replacing the old llvmpipe-crash bullet).

* Prepare 0.19.1 release

* State the desktop LiteRT-LM embeddings runtime floor for 0.19.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant