Skip to content

Sweep #1950: grammar-pause wording, lazy-grammar test, drift gate, WebGPU error map - #1965

Merged
leehack merged 1 commit into
mainfrom
lane-1950-final
Nov 12, 2026
Merged

leehack merged 1 commit into
mainfrom
lane-1950-final

Conversation

@leehack

@leehack leehack commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Summary

Tracks #1950 (does not close it). Four small audit leftovers before 0.19.8 release prep:

Production-readiness scope

  • User-facing scope: accurate docs and error text for thinking-budget grammar handling; WebGPU thinking-budget rejections surface as LlamaUnsupportedException.
  • Supported platforms/paths: native llama.cpp (test only), WebGPU (error mapping, message), docs and release tooling.
  • Unsupported or intentionally unavailable paths: unchanged; assets without thinkingBudgetPauseGrammar still throw LlamaUnsupportedException.
  • Out of scope / follow-ups: LiteRT-LM e2e skip, Streaming partial parses differ from the final parse by chunking (MiniMax M3, thinking tags, Gemma 4) #1930, llama-web-bridge items. Shard weights (see below): no change.

Completeness checklist

  • Declared scope is fully implemented, or explicitly reduced above.
  • Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable.
  • Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant (existing Keep response-format grammars applied through a budgeted thought on WebGPU (#1954) #1964 bullets reworded; no new bullet).
  • Regression coverage covers the original issue plus key negative/version-skew paths where relevant.
  • Public capability changes update the validation coverage registry: N/A, no capability change.
  • Security/privacy review completed: no secrets or URLs touched.
  • Follow-up work tracked: None.

PR type guidance

Test Plan

  • dart run tool/prepare_workspace.dart
  • dart format --output=none --set-exit-if-changed .
  • dart analyze
  • dart test -p vm -j 1 --exclude-tags local-only: touched VM tests (test/integration/thinking_budget_grammar_test.dart with TMPDIR isolated and 2 threads, test/unit/tooling/, all pass); full lane left to CI
  • dart test -p chrome --exclude-tags local-only: test/unit/backends/webgpu/ (540 pass); full lane left to CI
  • coverage-lib matrix row: N/A, runtime lib/ behavior unchanged apart from an error-mapping branch and message text covered by the Chrome test
  • Other: check_platform_boundaries, check_webgpu_bridge_tag.dart --verify-manifest, verify_release_docs_versions.dart plain (pass) and --release-prep (fails only on the expected missing ## 0.19.7 heading and kept ## Unreleased, both changelogs), ./tool/docs/build_site.sh

Mutation results

# Item Mutation Result
2 Lazy-grammar test never apply a lazy grammar sampler in llama_cpp_service.dart new test fails; the three older tests still pass (the gap)
3a Drift message corrupt the "first differing line" text killed
3b Docs-page wording drop the * marker check / drop the "the default" check each killed
4 WebGPU errors drop the thinkingBudget.pauseGrammar alternative / drop the thinkingBudget gate each killed (Chrome)
1 Wording docs and Dartdoc only; the ## Unreleased bullets are exempt from the drift gate, so no test can pin them N/A

Shard weights (#1964 audit T7/T7b): not changed. doc/ci_selection.md makes a missing entry harmless (a new file counts as the median and stale weights only unbalance shards), 29 test files already lack an entry, and the two new entries are placeholders of 0. An "every file has an entry" rule would contradict that contract and fail today; an "every entry exists" rule would only add friction when tests are deleted.

Matrix Evidence

Matrix row Scope covered Platform / model / backend Result Evidence / notes
high-risk-exact-head-independent-qa independent blocking-only review of exact head macOS, stories15M, Chrome PASS fresh auditor-deep at b9569fb; 17/18 mutants killed (E4 $ anchor predates the PR); 0.19.8 release-prep simulation passes
release-metadata-verification release docs verifier and drift gate VM PASS verify_release_docs_versions.dart plain and --release-prep as above; test/unit/tooling/ pass
structured-output-adversarial grammar/structured output N/A N/A no grammar compilation or acceptance change; one test added
litert-lm-v1-public-direct, litert-lm-v1-public-service, litert-lm-v1-public-speech, litert-lm-json-schema, chat-app-litert-json-schema LiteRT-LM N/A N/A LiteRT-LM untouched

Review Notes

High-risk regression review

  • Classification: high-risk (backendRuntime, releaseAutomation: webgpu_errors.dart, webgpu_backend.dart message, verify_release_docs_versions.dart)
  • Implementation task: this lane (worker)
  • Independent blocking QA task: fresh auditor-deep, PASS. Simulated 0.19.8 release prep exits 0 and a negative control fails. Evaluator exit 2 at head b9569fba / base c2eb1a82. CI: 32 pass, 9 skipped.
  • Exact head / current base: b9569fba60a0075b210f8934e6f8aad648fd4b96 / c2eb1a822d
  • Production-branch deletion, bypass, or miswire proof: N/A, no production branch removed; error mapping verified by Chrome test and mutation
  • Affected-family real-model/artifact evidence: the lazy-grammar test runs on the stories15M model; no model family behavior changed
  • Explicit unavailable-family or other N/A evidence: N/A
  • Known PR-caused P1 regressions: 0
  • Unresolved review threads: 0

@leehack
leehack marked this pull request as ready for review November 12, 2026 20:52
@leehack
leehack merged commit 63fb1ec into main Nov 12, 2026
43 checks passed
@leehack
leehack deleted the lane-1950-final branch November 12, 2026 20:53
@github-actions

Copy link
Copy Markdown
Contributor

Chat app preview removed for leehack/llamadart-chat-pr-1965.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant