[leehack]

Summary

• Native LiteRT-LM generate() now sends the prompt unchanged. It used to create its conversation with the bundle's chat template, so the runtime wrapped the prompt in a user turn and generation prompt (12 extra tokens on Qwen3.5, 8 on Qwen3, 7 on Gemma 4).
• create() requests that fall back to a Dart-rendered prompt (a trailing system message, parallelToolCalls: true) reach the same backend method, so they were templated twice. They are now templated once.
• How: raw generation creates its conversation with liteRtLmRawPromptTemplate, an internal template that copies the message text through. It uses the existing litert_lm_conversation_config_set_prompt_template setter, so no native change is needed. Raw generation no longer runs the bundle-template probe, and its send failures no longer carry the bundle-template note.
• Start token, as llama.cpp handles it: the runtime adds the model's start token (BOS) before every prompt. A raw prompt that already starts with it is sent without it, so it doesn't get two.
  • Special-token text such as Gemma's <bos> (which the runtime parses as the token but the tokenizer API does not) is removed as text.
  • Ordinary text that tokenizes to the start token (Qwen3-0.6B's start token is the word None) is removed only when the rest of the prompt keeps its tokens.
  • The start token is read once per model, through tokenize('', addSpecial: true) and detokenize.
• Empty prompt: an empty raw prompt, or one that is only the start token, now throws LlamaArgumentException(name: 'prompt') before native work. Previously the runtime failed with "No token IDs found" (Qwen3.5) or "Session is not prefilled yet" (Gemma).
• Legacy runtimes: a pre-C-API-1.0 runtime without the template setter throws LlamaUnsupportedException. The message names the setter, says llamadart cannot set a conversation template (needed by raw prompt generation, ModelParams.chatTemplate and the built-in family templates such as Qwen3 text chat), and names the packaged litert-lm-native version.
• Chat app: the example's LiteRT-LM benchmark (litert_lm_benchmark_app.dart) now measures a user turn through create() rather than raw generate(), so its numbers still describe chat prompts.
• Docs: a note in generation-and-streaming.md, the llamadart-chat-streaming SKILL line, and a changelog bullet in both files.

Fixes #1853. Refs #1845 (lane 9). Web follow-up: #1863. Pre-existing bug found by the audit, not fixed here: #1867.

Stack: based on #1854 (lane3-1836-template-split, audited at d6a62a80). Merge #1854 first; this PR's diff is only #1853.

Maintainer decision (option A of the options in #1853): "Raw generate (Recommended)". Options considered: A, make generate() raw; B, keep the wrapping, document it, and add new backend SPI so only the fallbacks go raw; C, refuse the fallbacks on LiteRT-LM.

Production-readiness scope

• User-facing scope: on native LiteRT-LM, engine.generate(prompt) passes the prompt to the model without a chat template, as on llama.cpp and as generate() is documented, with exactly one start token where the model has one. create() fallbacks send the Dart-rendered prompt once.
• Supported platforms/paths: native LiteRT-LM on every native platform with the packaged v0.19.1-2 runtime (C API 1.0, which requires the setter), plus legacy runtimes that export it.
• Unsupported or intentionally unavailable paths:
  • An empty raw prompt throws LlamaArgumentException.
  • A legacy runtime without the template setter throws LlamaUnsupportedException for raw generation instead of silently wrapping the prompt.
  • LiteRT-LM Web still sends the raw prompt as one user turn: @litert-lm/[email protected] cannot replace its conversation template. This is documented in the guide and SKILL and tracked in LiteRT-LM Web generate() wraps the raw prompt in the bundle's user turn #1863. Web create() is not double-templated, because its Dart template passes only the latest message's text through.
• Out of scope / follow-ups: LiteRT-LM Web generate() wraps the raw prompt in the bundle's user turn #1863 (Web) and LiteRT-LM bundle-template probe lets a C API 1.0 render failure escape, so chats on string-concatenating templates fail #1867 (pre-existing bundle-template probe bug).

Completeness checklist

• Declared scope is fully implemented, or explicitly reduced above.
• Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable.
• Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant. (No README or support-matrix mention; the example benchmark is updated.)
• Regression coverage covers the original issue plus key negative/version-skew paths where relevant.
• Public capability changes update the validation coverage registry with a pack or a reasoned, issue-linked exemption; inventory checks pass (or N/A with scope). N/A: no capability change; validation_coverage_test.dart passes in the Markdown suites.
• Security/privacy review completed: no secrets, bearer tokens, signed URLs, or raw secret-bearing paths leak through logs, cache keys, metadata, errors, or snapshots. New error texts name a symbol and a version only.
• Follow-up work that is useful but not required for this PR is tracked in GitHub Issues, or explicitly marked "None" above.

PR type guidance

• Feature PR: N/A
• Bugfix PR: Root cause: LiteRtLmService._generate built its conversation through _createBundleTemplateConversation, and the client sends the prompt as {"role":"user","content":[{"type":"text","text":prompt}]}. The bundle template therefore wrapped every raw prompt, including the fallback prompts create() had already rendered. Regression tests are listed below.
• Docs-only PR: N/A

Test Plan

• dart run tool/prepare_workspace.dart
• dart format --output=none --set-exit-if-changed .
• dart analyze (no issues); check_platform_boundaries.dart OK
• dart test -p vm -j 1 --exclude-tags local-only: CI run 42660841299 green (workflow_dispatch; CI does not run on PRs into a non-default base). Locally, test/unit/backends/litert_lm/ plus the Markdown suites pass (2247 tests).
• dart test -p chrome --exclude-tags local-only: Web (Chrome) 1–3 in the same CI run (no Web code changed)
• coverage-lib: 93.63% line coverage (threshold 70) at 6101b292, local macOS arm64. The follow-up commit adds only covered code (every new branch is hit by the unit tests, as the mutations show). The local run's 6 failures are all in windows_registry_qualification_test.dart, which needs pwsh (missing on this host); they pass on CI Windows tooling.
• Chat app: (cd example/chat_app && flutter test test test_vm) passes (690), and the Chrome subset (chat_generation_service_test, image_generation_screen_test, unit_test, conversation_state_service_test) passes (133).
• Other targeted/local-only validation: see the matrix.

Failing-first:

• First commit, run against the base litert_lm_service.dart and litert_lm_runtime.dart (keeping the new constant so the tests compile): 4 tests fail. The new local-only e2e test also fails on base: Qwen3 runtime input 10 vs 2 expected; Qwen3.5 13 vs 1.
• Audit follow-ups, run against 6101b292's service and runtime: 5 tests fail (the start-token pair, the two empty-prompt refusals, and the legacy wording).

New and changed tests:

• litert_lm_service_test.dart:
  • Raw generation creates one conversation with liteRtLmRawPromptTemplate, passes the prompt unchanged, and neither reads the bundle nor runs the probe; a later chat still probes and adapts.
  • A runtime refusal of the template propagates and nothing is sent.
  • A failed raw send keeps its error.
  • New group "raw prompt start token":
    • special start text is dropped only when it leads the prompt;
    • plain start text is dropped only when the rest keeps its tokens;
    • nothing is dropped when the runtime adds no start token;
    • an empty prompt, or one that is only the start token, throws LlamaArgumentException before any conversation or send.
  • The 11 bundle-template probe tests that used raw generate as their trigger now use generateChat.
• litert_lm_bundle_template_test.dart: the template renders prompts unchanged as parts and as a string. The prompts include leading and trailing whitespace, ChatML, literal Jinja syntax and Unicode. It renders only text parts.
• litert_lm_conversation_template_test.dart: the legacy refusal names the setter, "cannot set a conversation template", raw prompt generation, and litert-lm-native $liteRtLmReleaseTag.
• test/e2e/backends/litert_lm_bundle_template_e2e_test.dart (local-only):
  • The real runtime renders raw prompts unchanged through the template.
  • Service raw generation sends each prompt with runtime input tokens equal to tokenize(prompt, addSpecial: true).
  • A prompt of the start-token text plus hi is sent as hi, with runtime input tokens equal to tokenize('hi', addSpecial: true), and the start token alone throws LlamaArgumentException.
  • All pass on gemma-4-E2B-it and Qwen3-0.6B. Qwen3.5-0.8B_int8 passes the first two; it has no start token, so the start-token test skips.

Mutations (each applied alone against the three unit files; all 14 killed):

Mutation	Killed by
Raw generation uses the bundle template again	sends-unchanged, refusal
Template drops string content	template render tests
Template trims text	template render tests
Legacy message omits the setter name	legacy refusal
Legacy message drops the general cause	legacy refusal
Template refusal falls back to the bundle template	refusal
Prompt trimmed before sending	sends-unchanged
No start-token stripping	start-token tests
Special start text kept	special start-token test
Plain start text dropped without the token check	plain start-token test
Start text dropped when not leading	special start-token test
Empty prompt reaches the runtime	empty-prompt refusals
Original prompt sent instead of the stripped one	start-token tests
Start text always treated as plain	special start-token test

The first commit's "raw send failure gets the bundle-template note" mutation was killed at 6101b292, and that code is unchanged.

Token counts. Native LiteRT-LM v0.19.1-2, macOS arm64 CPU, numberOfThreads: 2. "Sent" is tokenize(prompt, addSpecial: true) of the prompt _generate received.

Bundle	Request	Sent	Runtime before	Runtime after
Qwen3.5-0.8B_int8	generate('hi')	1	13	1
Qwen3.5-0.8B_int8	generate(ChatML 'hi' turn)	9	21	9
Qwen3.5-0.8B_int8	create(), trailing system message, tools	155	167	155
Qwen3.5-0.8B_int8	create(parallelToolCalls: true), tools	147	159	147
Qwen3-0.6B	generate('hi')	2	10	2
Qwen3-0.6B	create(), trailing system message, tools	150	158	150
Qwen3-0.6B	create(parallelToolCalls: true), tools	142	150	142
gemma-4-E2B-it	generate('hi')	2	10	2
gemma-4-E2B-it	generate('<bos>hi')	2	3 at 6101b292	2
gemma-4-E2B-it	generate(chatTemplate().prompt), which starts with <bos>	16	17 at 6101b292	16
gemma-4-E2B-it	create(), trailing system message, tools	63	72	63
gemma-4-E2B-it	create(parallelToolCalls: true), tools	55	64	55

Native-chat create() is unchanged (Qwen3.5 16/16, Qwen3 142/142, Gemma 57/57). For the Gemma <bos> rows, "Sent" is the prompt's token count with its leading <bos> read as the single start token.

Matrix Evidence

Matrix row	Scope covered	Platform / model / backend	Result	Evidence / notes
root-vm	VM unit/integration	Linux, macOS, Windows CI	PASS	CI run 42660841299
coverage-lib	lib coverage threshold	macOS arm64 local	PASS	93.63% (33301/35566) at 6101b292
litert-lm-engine-smoke	raw generation through the public engine	macOS arm64 CPU, Qwen3-0.6B and Qwen3.5-0.8B_int8	PASS (at 6101b292)	promptEvalTokens 14 = promptTokenCountWithSpecial 14 (Qwen3); 13 = 13 (Qwen3.5)
litert-lm-chat-features-smoke	chat, cutoff, thinking, text history, structured output, tool gating	macOS arm64 auto, Qwen3-0.6B and Qwen3.5-0.8B_int8	PASS (at 6101b292)	Qwen3.5 thinking N/A (template pre-closes <think>); the follow-up does not touch chat
chat app VM + Chrome subset	benchmark app change	macOS host	PASS	690 VM, 133 Chrome
bundle-template e2e (local-only)	raw prompt and start token reach the runtime as intended	Qwen3.5-0.8B_int8, Qwen3-0.6B, gemma-4-E2B-it, CPU	PASS (new tests)	The suite's other failures exist on main too; see #1867
high-risk-exact-head-independent-qa	independent audit	see Review Notes	PASS at 6101b292; re-audit of the new head requested
litert-lm-v1-public-*, litert-lm-json-schema			N/A	No C API 1 adoption or schema change

Review Notes

• Independent review status: the independent audit of 6101b292 passed with no blocking findings. Its P2 (double start token) and P3s (empty prompt, legacy wording, benchmark) are fixed in ee80158a. It also found LiteRT-LM bundle-template probe lets a C API 1.0 render failure escape, so chats on string-concatenating templates fail #1867, which is pre-existing on main and filed. The head has moved, so the audit must be re-run or extended to ee80158a before mark-ready.
• CI status / head SHA: ee80158ada816bd8cec329afa16c3daf20e24958, CI run 42660841299 green (workflow_dispatch; CI does not run on PRs into a non-default base).

High-risk regression review

• Classification: high-risk (backendRuntime)
• Implementation task: lane 9 of 0.19.2 release plan: audit follow-ups and correctness fixes #1845
• Independent blocking QA task: passed at 6101b292 (no blocking); re-audit of ee80158a pending
• Exact head / current base: ee80158ada816bd8cec329afa16c3daf20e24958 / lane3-1836-template-split at d6a62a80
• Production-branch deletion, bypass, or miswire proof: 14 mutations, all killed (raw generation back on the bundle template, refusal fallback, prompt rewrite, start-token and empty-prompt branches)
• Affected-family real-model/artifact evidence: Qwen3.5-0.8B_int8, Qwen3-0.6B, gemma-4-E2B-it on the real runtime (token table, e2e, smokes)
• Explicit unavailable-family or other N/A evidence: LiteRT-LM Web not changed (LiteRT-LM Web generate() wraps the raw prompt in the bundle's user turn #1863); GPU/NPU not run (the template and start-token paths do not depend on the backend)
• Known PR-caused P1 regressions: 0
• Unresolved review threads: 0

[leehack]

Landed through the 0.19.2 batch integration PR #1870 (rebase merge, one commit per constituent; this PR's pinned head was audited/CI-gated there).