From e017646a15a05fca130882ee8c7e57762afb4428 Mon Sep 17 00:00:00 2001 From: leehack Date: Thu, 12 Nov 2026 13:31:12 -0500 Subject: [PATCH 1/2] Keep response-format grammars applied through a budgeted thought on WebGPU (#1954) Repin bridge assets v0.1.83 -> v0.1.84, which reports thinkingBudgetPauseGrammar. Send thinkingBudget.pauseGrammar: false only for an eager grammar that keepsAppliedThroughThought, as native does, and throw LlamaUnsupportedException on assets without the capability. --- CHANGELOG.md | 1 + README.md | 2 +- doc/multimodal_embeddings_web.md | 2 +- doc/testing_matrix.md | 7 + doc/webgpu_bridge.md | 25 ++- example/chat_app/web/index.html | 2 +- lib/src/backends/webgpu/interop.dart | 9 + lib/src/backends/webgpu/webgpu_backend.dart | 30 +++- .../models/inference/generation_params.dart | 4 +- lib/src/web_setup/web_bridge_setup.dart | 4 +- scripts/fetch_webgpu_bridge_assets.sh | 6 +- skills/llamadart-web/SKILL.md | 4 + .../thinking_budget_grammar_e2e_test.dart | 170 ++++++++++++++++++ test/support/fake_webgpu_feature_bridge.dart | 1 + .../webgpu_thinking_budget_grammar_test.dart | 117 ++++++++++++ .../tooling/check_webgpu_bridge_tag_test.dart | 72 ++++---- test/unit/tooling/test_matrix_test.dart | 1 + tool/ci/test_shard_weights.json | 8 + tool/testing/check_webgpu_bridge_tag.dart | 6 +- tool/testing/run_local_e2e.dart | 10 +- tool/testing/test_matrix.dart | 4 +- website/docs/changelog/recent-releases.md | 1 + website/docs/platforms/webgpu-bridge.md | 16 +- 23 files changed, 436 insertions(+), 66 deletions(-) create mode 100644 test/e2e/webgpu/thinking_budget_grammar_e2e_test.dart create mode 100644 test/unit/backends/webgpu/webgpu_thinking_budget_grammar_test.dart diff --git a/CHANGELOG.md b/CHANGELOG.md index d87f8ff9a..f6e4abcd0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ Releases before 0.10.0 are in the - Fixed constrained JSON (`responseFormat`) ignoring the grammar after a `ThinkingBudget` closes a template-opened thought on native llama.cpp (Qwen3.5 and similar), and the same for required tool calls on templates whose grammar takes that thought. - Fixed a required tool call on Seed-OSS being unconstrained after a `ThinkingBudget` closes a thought the model opened, on native llama.cpp ([#1958](https://github.com/leehack/llamadart/issues/1958)). +- Aligned the default WebGPU bridge assets to `v0.1.84`: constrained JSON (`responseFormat`) and Seed-OSS required tool calls stay constrained after a `ThinkingBudget` closes a thought on WebGPU, and older assets throw `LlamaUnsupportedException` for that combination ([#1954](https://github.com/leehack/llamadart/issues/1954)). Web keeps native llama.cpp `v0.9.0` parity (upstream `v0.9.0@cd959bae99ff4bed22bef2604f45241c09bbb8c8`) and `@litert-lm/core@0.19.0`. Manifest SHA-256: `55abeb377597148156059a8ee5400f2314c6254ad91b57db65b709e304c0dea0`. ## 0.19.7 diff --git a/README.md b/README.md index d0bf9fe27..38796895c 100644 --- a/README.md +++ b/README.md @@ -164,7 +164,7 @@ Current default runtime pins: | --- | --- | | Native llama.cpp / GGUF | `leehack/llamadart-native@v0.9.0` | | Native LiteRT-LM / `.litertlm` | `leehack/litert-lm-native@v0.19.1-4` | -| Web llama.cpp / GGUF | `leehack/llama-web-bridge-assets@v0.1.83` | +| Web llama.cpp / GGUF | `leehack/llama-web-bridge-assets@v0.1.84` | | Web LiteRT-LM / `.litertlm` | `@litert-lm/core@0.19.0` | Native LiteRT-LM uses a lazy ABI-aware client for direct calls, services and diff --git a/doc/multimodal_embeddings_web.md b/doc/multimodal_embeddings_web.md index bb32bccf8..e4e503422 100644 --- a/doc/multimodal_embeddings_web.md +++ b/doc/multimodal_embeddings_web.md @@ -39,7 +39,7 @@ Bridge source contract inspected read-only at restore and structured-clone ownership; those are bridge responsibilities. Qualification target: EmbeddingGemma-2 BF16 with its matching -mmproj and pinned assets `v0.1.83` on Chrome CPU wasm memory64 +mmproj and pinned assets `v0.1.84` on Chrome CPU wasm memory64 (`gpuLayers: 0`). The scenario below checks the bridge's production worker budget against references from the matching native `v0.9.0`. The table below retains the original `v0.1.66` lineage diff --git a/doc/testing_matrix.md b/doc/testing_matrix.md index e2024702f..129a3df35 100644 --- a/doc/testing_matrix.md +++ b/doc/testing_matrix.md @@ -1452,6 +1452,13 @@ prompt token, verifies unchanged ordinary grammar defaults, and verifies that explicit prompt penalties change the pick without activating a prompt-matching lazy trigger. Keep both memory64 assets beside the wasm32 files. +The thinking-budget control adds `"reasoningModel"` (Qwen3.5-0.8B Q4_K_M) and, +for the version-skew control, `"legacyAssets"` (an earlier bridge directory +without `thinkingBudgetPauseGrammar`, such as `v0.1.83`) to the config. Qwen3.5 +`responseFormat` with a `ThinkingBudget` of 0, 1 and 16 tokens must still return +the schema's JSON in both modes, and the legacy assets must throw +`LlamaUnsupportedException` for a budget. + This qualifies the Web grammar plumbing on the selected weights, not hardware WebGPU or LFM2 auto accuracy/enforcement (#1715). The latter still needs its separate template contract and false-probe restriction. diff --git a/doc/webgpu_bridge.md b/doc/webgpu_bridge.md index 77240a9b4..25d1597fe 100644 --- a/doc/webgpu_bridge.md +++ b/doc/webgpu_bridge.md @@ -67,11 +67,11 @@ an earlier ` ``` That release embeds llama.cpp `v0.9.0`, matching the `hook/build.dart` native pin (`v0.9.0`, both built from upstream llama.cpp `v0.9.0@cd959bae99ff4bed22bef2604f45241c09bbb8c8`) -even though the bridge asset tag `v0.1.83` differs from the native runtime tag -`v0.9.0`. Provenance for this immutable consumer artifact: release `438242670`, -tag commit `44aa84a8a51701dc9c4f82c21e6ee3e774919c20`, bridge source -`cb6383fbe7e903764a89f5c80c7f81c219f99769`, and manifest SHA-256 -`ae6f488408672f9ba28d8f9e664a2b85f4e12048f290e259e18dd0b750c038e5`. +even though the bridge asset tag `v0.1.84` differs from the native runtime tag +`v0.9.0`. Provenance for this immutable consumer artifact: release `443341300`, +tag commit `0a3c45f24fd6293050e1772bed4a425b2b5a8493`, bridge source +`dc359398bc6445ec15878ddb873555295f957f85`, and manifest SHA-256 +`55abeb377597148156059a8ee5400f2314c6254ad91b57db65b709e304c0dea0`. They expose the API-version-1 media embedding contract. State and extraction qualification covers wasm32 and memory64 through direct and worker runtimes. Existing decision, scoring, sampling, LoRA, speculative decoding and speech APIs remain available. @@ -205,6 +205,15 @@ physical playback, intelligibility or speaker-reference fidelity. suppress tokens, so Qwen3-TTS no longer fails with `code out of range` when it samples a codec control token ([llama-web-bridge#211](https://github.com/leehack/llama-web-bridge/pull/211)). +- `v0.1.84+`: `getCompletionCapabilities()` reports + `thinkingBudgetPauseGrammar`, and `thinkingBudget.pauseGrammar: false` keeps a + grammar applied while the model reasons + ([llama-web-bridge#213](https://github.com/leehack/llama-web-bridge/pull/213)). + llamadart sends it, as native does, only when `grammarLazy` is off and the + grammar takes the thought (`ToolCallGrammarUtils.keepsAppliedThroughThought`: + constrained JSON on Qwen3.5, required tool calls on Seed-OSS), and throws + `LlamaUnsupportedException` for that combination on assets without the flag. + Every other request omits `pauseGrammar` and is unchanged. ### Media embeddings diff --git a/example/chat_app/web/index.html b/example/chat_app/web/index.html index 17cd13fc9..6dcaae91a 100644 --- a/example/chat_app/web/index.html +++ b/example/chat_app/web/index.html @@ -157,7 +157,7 @@ typeof configuredRepo === 'string' && configuredRepo.length > 0 ? configuredRepo : 'leehack/llama-web-bridge-assets'; - const defaultBridgeAssetsTag = 'v0.1.83'; + const defaultBridgeAssetsTag = 'v0.1.84'; const defaultBridgeLlamaCppTag = 'v0.9.0'; const bridgeAssetsTag = typeof configuredTag === 'string' && configuredTag.length > 0 diff --git a/lib/src/backends/webgpu/interop.dart b/lib/src/backends/webgpu/interop.dart index 4dd67bd2d..4e5bf80b1 100644 --- a/lib/src/backends/webgpu/interop.dart +++ b/lib/src/backends/webgpu/interop.dart @@ -299,6 +299,11 @@ extension type WebGpuThinkingBudgetOptions._(JSObject _) implements JSObject { @JS('endTag') required String endTag, @JS('forcedMessage') required String forcedMessage, }); + + /// `false` keeps the grammar applied inside a reasoning block; set only + /// when the loaded core reports `thinkingBudgetPauseGrammar`. + @JS('pauseGrammar') + external set pauseGrammar(bool value); } /// Completion options reported by `getCompletionCapabilities`. @@ -317,6 +322,10 @@ extension type WebGpuCompletionCapabilities._(JSObject _) implements JSObject { @JS('thinkingBudget') external JSAny? get thinkingBudget; + /// Whether the loaded core applies `thinkingBudget.pauseGrammar`. + @JS('thinkingBudgetPauseGrammar') + external JSAny? get thinkingBudgetPauseGrammar; + /// Whether the loaded core counts the prompt in its penalty window and /// applies `penaltyPromptTokens`. @JS('penaltyPromptTokens') diff --git a/lib/src/backends/webgpu/webgpu_backend.dart b/lib/src/backends/webgpu/webgpu_backend.dart index d093a9c7f..463e22c41 100644 --- a/lib/src/backends/webgpu/webgpu_backend.dart +++ b/lib/src/backends/webgpu/webgpu_backend.dart @@ -23,6 +23,7 @@ import '../../core/models/inference/model_params.dart'; import '../../core/models/inference/model_params_runtime_support.dart'; import '../../core/models/inference/next_token_scores.dart'; import '../../core/template/media_placeholders.dart'; +import '../../core/template/tool_call_grammar_utils.dart'; import '../backend.dart'; import '../model_params_loras.dart'; import '../web/webgpu_adapter_probe.dart'; @@ -43,6 +44,7 @@ typedef _WebGpuCompletionCapabilities = ({ bool presencePenalty, bool minP, bool thinkingBudget, + bool thinkingBudgetPauseGrammar, bool penaltyPromptTokens, bool penaltyPromptTokensWithGrammar, bool lazyGrammar, @@ -99,6 +101,7 @@ class WebGpuLlamaBackend presencePenalty: false, minP: false, thinkingBudget: false, + thinkingBudgetPauseGrammar: false, penaltyPromptTokens: false, penaltyPromptTokensWithGrammar: false, lazyGrammar: false, @@ -1087,6 +1090,9 @@ class WebGpuLlamaBackend presencePenalty: reported(capabilities.presencePenalty), minP: reported(capabilities.minP), thinkingBudget: reported(capabilities.thinkingBudget), + thinkingBudgetPauseGrammar: reported( + capabilities.thinkingBudgetPauseGrammar, + ), penaltyPromptTokens: reported(capabilities.penaltyPromptTokens), penaltyPromptTokensWithGrammar: reported( capabilities.penaltyPromptTokensWithGrammar, @@ -1941,6 +1947,22 @@ class WebGpuLlamaBackend 'assets do not.', ); } + final grammar = params.grammar; + final keepGrammarApplied = + thinkingBudget != null && + grammar != null && + !params.grammarLazy && + ToolCallGrammarUtils.keepsAppliedThroughThought(grammar); + if (keepGrammarApplied && !capabilities.thinkingBudgetPauseGrammar) { + throw LlamaUnsupportedException( + 'WebGPU thinking-budget control with a grammar that stays applied ' + 'through the thought (responseFormat on Qwen3.5, required tool calls ' + 'on Seed-OSS) needs bridge assets v0.1.84+ whose ' + 'getCompletionCapabilities() reports thinkingBudgetPauseGrammar; ' + 'the loaded core pauses every grammar while the model reasons, so ' + 'the reply would not be constrained.', + ); + } final speculativeConfig = params.resolvedSpeculativeDecodingConfig; if (speculativeConfig != null) { requireWebGpuSpeculativeSupport( @@ -1992,6 +2014,7 @@ class WebGpuLlamaBackend : _bridgeThinkingBudget( thinkingBudget, hasMediaParts: mediaParts != null, + pauseGrammar: !keepGrammarApplied, ); final speculative = resolveWebGpuSpeculativeRequest( params, @@ -2319,6 +2342,7 @@ class WebGpuLlamaBackend WebGpuThinkingBudgetOptions _bridgeThinkingBudget( ThinkingBudget budget, { required bool hasMediaParts, + required bool pauseGrammar, }) { if (budget.maxTokens < 0 || budget.maxTokens > _maxThinkingBudgetTokens) { throw RangeError.range( @@ -2347,12 +2371,16 @@ class WebGpuLlamaBackend 'selected chat template automatically.', ); } - return WebGpuThinkingBudgetOptions( + final options = WebGpuThinkingBudgetOptions( maxTokens: budget.maxTokens, startTag: startTag, endTag: endTag, forcedMessage: budget.forcedMessage ?? '', ); + // Sent only to keep a grammar applied, so every other request is + // unchanged on the wire. + if (!pauseGrammar) options.pauseGrammar = false; + return options; } @override diff --git a/lib/src/core/models/inference/generation_params.dart b/lib/src/core/models/inference/generation_params.dart index 5b7bc071f..34348d8a0 100644 --- a/lib/src/core/models/inference/generation_params.dart +++ b/lib/src/core/models/inference/generation_params.dart @@ -956,7 +956,9 @@ class GenerationParams { /// models the thought the model opens (Seed-OSS's required-call root, /// starting with `seed-thought`) applied while budgeted reasoning is /// active. It pauses every other grammar, lazy grammars included, until - /// reasoning has ended. Native LiteRT-LM applies it to chat generation on + /// reasoning has ended. WebGPU does the same on bridge assets that report + /// `thinkingBudgetPauseGrammar` (v0.1.84+) and rejects the keep-applied + /// grammars on older assets. Native LiteRT-LM applies it to chat generation on /// bundles with a thinking channel, which it closes with the bundle's own /// end marker; it rejects a zero budget and [ThinkingBudget.forcedMessage]. /// Other backends reject this setting rather than silently ignoring it. diff --git a/lib/src/web_setup/web_bridge_setup.dart b/lib/src/web_setup/web_bridge_setup.dart index cd40a18da..4f7d18740 100644 --- a/lib/src/web_setup/web_bridge_setup.dart +++ b/lib/src/web_setup/web_bridge_setup.dart @@ -15,14 +15,14 @@ import '../core/exceptions.dart'; const String webBridgeAssetsRepository = 'leehack/llama-web-bridge-assets'; /// The bridge asset release this llamadart version is qualified against. -const String webBridgeAssetsTag = 'v0.1.83'; +const String webBridgeAssetsTag = 'v0.1.84'; /// SHA-256 of the exact approved `manifest.json` of [webBridgeAssetsTag]. /// /// The manifest lists the SHA-256 and size of every asset, so pinning its /// bytes pins every file the tool writes. const String webBridgeManifestSha256 = - 'ae6f488408672f9ba28d8f9e664a2b85f4e12048f290e259e18dd0b750c038e5'; + '55abeb377597148156059a8ee5400f2314c6254ad91b57db65b709e304c0dea0'; /// Directory under the app's `web/` directory that receives the assets. const String webBridgeAssetsDirName = 'webgpu_bridge'; diff --git a/scripts/fetch_webgpu_bridge_assets.sh b/scripts/fetch_webgpu_bridge_assets.sh index f1bb10215..5a09b3b04 100755 --- a/scripts/fetch_webgpu_bridge_assets.sh +++ b/scripts/fetch_webgpu_bridge_assets.sh @@ -4,7 +4,7 @@ set -euo pipefail ROOT_DIR="$(git rev-parse --show-toplevel)" OUT_DIR="${WEBGPU_BRIDGE_OUT_DIR:-$ROOT_DIR/example/chat_app/web/webgpu_bridge}" ASSETS_REPO="${WEBGPU_BRIDGE_ASSETS_REPO:-leehack/llama-web-bridge-assets}" -ASSETS_TAG="${WEBGPU_BRIDGE_ASSETS_TAG:-v0.1.83}" +ASSETS_TAG="${WEBGPU_BRIDGE_ASSETS_TAG:-v0.1.84}" CDN_BASE="${WEBGPU_BRIDGE_CDN_BASE:-https://cdn.jsdelivr.net/gh/${ASSETS_REPO}@${ASSETS_TAG}}" CURL_DOWNLOAD_ARGS=( --connect-timeout 15 @@ -19,7 +19,7 @@ if [[ "${1:-}" == "--help" || "${1:-}" == "-h" ]]; then Downloads prebuilt WebGPU bridge assets into the chat_app web directory. Default source: - https://cdn.jsdelivr.net/gh/leehack/llama-web-bridge-assets@v0.1.83 + https://cdn.jsdelivr.net/gh/leehack/llama-web-bridge-assets@v0.1.84 Environment variables: WEBGPU_BRIDGE_ASSETS_REPO Asset repo in owner/repo format @@ -31,7 +31,7 @@ Usage: ./scripts/fetch_webgpu_bridge_assets.sh Examples: - WEBGPU_BRIDGE_ASSETS_TAG=v0.1.83 ./scripts/fetch_webgpu_bridge_assets.sh + WEBGPU_BRIDGE_ASSETS_TAG=v0.1.84 ./scripts/fetch_webgpu_bridge_assets.sh WEBGPU_BRIDGE_ASSETS_REPO=acme/llama-web-bridge-assets WEBGPU_BRIDGE_ASSETS_TAG=v2 ./scripts/fetch_webgpu_bridge_assets.sh USAGE exit 0 diff --git a/skills/llamadart-web/SKILL.md b/skills/llamadart-web/SKILL.md index 463e0b6e9..a256e8c37 100644 --- a/skills/llamadart-web/SKILL.md +++ b/skills/llamadart-web/SKILL.md @@ -101,6 +101,10 @@ in the llamadart-getting-started and llamadart-chat-streaming skills. unless `withTextHistory()` is used and the loaded runtime supports them. Unreported controls throw `LlamaUnsupportedException`; `engine.runtime` names the runtime, so never sniff the `.litertlm` extension after loading. + - A `ThinkingBudget` with a grammar that stays applied through the thought + (`responseFormat` on Qwen3.5, required calls on Seed-OSS) needs assets that + report `thinkingBudgetPauseGrammar`; older assets throw + `LlamaUnsupportedException`. - `engine.supportsNextTokenScoring` and `engine.supportsStatePersistence`. - Runtime LoRA calls, and loads with `ModelParams.loras`, throw diff --git a/test/e2e/webgpu/thinking_budget_grammar_e2e_test.dart b/test/e2e/webgpu/thinking_budget_grammar_e2e_test.dart new file mode 100644 index 000000000..b5a94a8d8 --- /dev/null +++ b/test/e2e/webgpu/thinking_budget_grammar_e2e_test.dart @@ -0,0 +1,170 @@ +/// Real pinned-asset check that a response-format grammar stays applied +/// through a budgeted forced-open thought on Qwen3.5 (#1954). +/// +/// Uses build/web_lazy_grammar/config.json, adding "reasoningModel" (a +/// Qwen3.5 GGUF) and, for the version-skew control, "legacyAssets" (bridge +/// assets without `thinkingBudgetPauseGrammar`, such as `v0.1.83`). Run the +/// web-lazy-grammar scenario. +@TestOn('browser') +@Tags(['local-only', 'e2e']) +@Timeout(Duration(minutes: 30)) +library; + +import 'dart:convert'; +import 'dart:js_interop'; +import 'dart:js_interop_unsafe'; + +import 'package:llamadart/llamadart.dart'; +import 'package:llamadart/src/backends/webgpu/interop.dart'; +import 'package:llamadart/src/backends/webgpu/webgpu_backend.dart'; +import 'package:test/test.dart'; +import 'package:web/web.dart' show window; + +const _messages = [ + LlamaChatMessage.fromText( + role: LlamaChatRole.user, + text: 'Give the capital of France as JSON {"city": ...}.', + ), +]; + +const _responseFormat = { + 'type': 'json_schema', + 'json_schema': { + 'schema': { + 'type': 'object', + 'properties': { + 'city': {'type': 'string'}, + }, + 'required': ['city'], + 'additionalProperties': false, + }, + }, +}; + +const _model = ModelParams( + contextSize: 4096, + gpuLayers: 0, + preferredBackend: GpuBackend.cpu, + numberOfThreads: 2, + numberOfThreadsBatch: 2, +); + +void main() { + final configUrl = Uri.base.resolve( + '../../../build/web_lazy_grammar/config.json', + ); + late Map config; + Uri fixture(String key) => configUrl.resolve(config[key] as String); + + Future bridgeConstructor(String assets) async { + final module = await importModule( + fixture(assets).resolve('llama_webgpu_bridge.js').toString().toJS, + ).toDart; + return module['LlamaWebGpuBridge']! as JSFunction; + } + + LlamaEngine newEngine(String assets, JSFunction constructor, bool worker) { + String asset(String name) => fixture(assets).resolve(name).toString(); + return LlamaEngine( + WebGpuLlamaBackend( + bridgeFactory: ([_]) => + constructor.callAsConstructor( + JSObject() + ..['disableWorker'] = (!worker).toJS + ..['workerUrl'] = asset('llama_webgpu_bridge_worker.js').toJS + ..['coreModuleUrl'] = asset('llama_webgpu_core.js').toJS + ..['wasmUrl'] = asset('llama_webgpu_core.wasm').toJS, + ), + ), + ); + } + + Future> run( + LlamaEngine engine, + ThinkingBudget? budget, + ) => engine + .create( + _messages, + params: GenerationParams( + maxTokens: 1536, + temp: 0, + seed: 42, + thinkingBudget: budget, + ), + enableThinking: true, + responseFormat: _responseFormat, + ) + .toList(); + + String content(List chunks) => + chunks.map((chunk) => chunk.choices.first.delta.content ?? '').join(); + + setUpAll(() async { + final response = await window.fetch(configUrl.toString().toJS).toDart; + expect(response.ok, isTrue, reason: 'Missing $configUrl'); + config = + jsonDecode((await response.text().toDart).toDart) + as Map; + expect( + config['reasoningModel'], + isNotNull, + reason: 'Stage "reasoningModel" in the config', + ); + }); + + for (final worker in [false, true]) { + test('${worker ? 'worker' : 'direct'}: constrained JSON survives a ' + 'budgeted thought', () async { + final engine = newEngine( + 'assets', + await bridgeConstructor('assets'), + worker, + ); + addTearDown(engine.dispose); + await engine.loadModelFromUrl( + fixture('reasoningModel').toString(), + modelParams: _model, + ); + expect((await engine.capabilities).supportsThinkingBudget, isTrue); + for (final tokens in [0, 1, 16]) { + final text = content( + await run(engine, ThinkingBudget(maxTokens: tokens)), + ); + expect(jsonDecode(text), { + 'city': isA(), + }, reason: 'budget=$tokens content=${jsonEncode(text)}'); + } + }); + } + + test('assets without thinkingBudgetPauseGrammar reject the budget instead ' + 'of leaving the reply unconstrained', () async { + if (config['legacyAssets'] == null) { + markTestSkipped('Stage "legacyAssets" for the version-skew control'); + return; + } + final engine = newEngine( + 'legacyAssets', + await bridgeConstructor('legacyAssets'), + false, + ); + addTearDown(engine.dispose); + await engine.loadModelFromUrl( + fixture('reasoningModel').toString(), + modelParams: _model, + ); + expect(jsonDecode(content(await run(engine, null))), { + 'city': isA(), + }); + await expectLater( + run(engine, const ThinkingBudget(maxTokens: 4)), + throwsA( + isA().having( + (error) => error.message, + 'message', + allOf(contains('thinkingBudgetPauseGrammar'), contains('v0.1.84+')), + ), + ), + ); + }); +} diff --git a/test/support/fake_webgpu_feature_bridge.dart b/test/support/fake_webgpu_feature_bridge.dart index bf443fb4e..eb121839f 100644 --- a/test/support/fake_webgpu_feature_bridge.dart +++ b/test/support/fake_webgpu_feature_bridge.dart @@ -236,6 +236,7 @@ class FakeFeatureBridge { 'presencePenalty', 'minP', 'thinkingBudget', + 'thinkingBudgetPauseGrammar', 'penaltyPromptTokens', 'penaltyPromptTokensWithGrammar', 'lazyGrammar', diff --git a/test/unit/backends/webgpu/webgpu_thinking_budget_grammar_test.dart b/test/unit/backends/webgpu/webgpu_thinking_budget_grammar_test.dart new file mode 100644 index 000000000..52630ff58 --- /dev/null +++ b/test/unit/backends/webgpu/webgpu_thinking_budget_grammar_test.dart @@ -0,0 +1,117 @@ +@TestOn('browser') +library; + +import 'package:llamadart/llamadart.dart'; +import 'package:llamadart/src/backends/webgpu/webgpu_backend.dart'; +import 'package:llamadart/src/core/template/tool_call_grammar_utils.dart'; +import 'package:test/test.dart'; + +import '../../../support/fake_webgpu_feature_bridge.dart'; + +void main() { + late FakeFeatureBridge fake; + late LlamaEngine engine; + + setUp(() { + fake = FakeFeatureBridge() + ..completionCapabilities = { + 'thinkingBudget': true, + 'thinkingBudgetPauseGrammar': true, + 'lazyGrammar': true, + }; + engine = LlamaEngine( + WebGpuLlamaBackend(bridgeFactory: ([_]) => fake.bridge), + ); + addTearDown(engine.dispose); + }); + + Future load() => + engine.loadModelFromUrl('https://example.com/tiny.gguf'); + Future generate(GenerationParams params) => + engine.generate('Once ', params: params).drain(); + final taking = ToolCallGrammarUtils.afterForcedOpenThought( + 'root ::= "ok"\n', + '', + ); + const budget = ThinkingBudget( + maxTokens: 8, + startTag: '', + endTag: '', + ); + const paused = { + 'maxTokens': 8, + 'startTag': '', + 'endTag': '', + 'forcedMessage': '', + }; + + const seedThought = 'root ::= seed-thought call\ncall ::= "x"\n'; + + test('keeps a grammar that takes or models the thought applied', () async { + await load(); + for (final grammar in [taking, seedThought]) { + await generate( + GenerationParams(grammar: grammar, thinkingBudget: budget), + ); + expect(fake.completionOption('thinkingBudget'), { + ...paused, + 'pauseGrammar': false, + }); + } + }); + + test('leaves every other grammar paused', () async { + await load(); + for (final params in [ + const GenerationParams(thinkingBudget: budget), + const GenerationParams(grammar: 'root ::= "ok"', thinkingBudget: budget), + GenerationParams( + grammar: taking, + grammarLazy: true, + grammarTriggers: const [ + GenerationGrammarTrigger.typed( + type: GrammarTriggerType.word, + value: 'a', + ), + ], + thinkingBudget: budget, + ), + ]) { + await generate(params); + expect(fake.completionOption('thinkingBudget'), paused); + } + }); + + test( + 'rejects a kept-applied grammar on assets that would pause it', + () async { + fake.completionCapabilities = {'thinkingBudget': true}; + await load(); + for (final grammar in [taking, seedThought]) { + await expectLater( + generate(GenerationParams(grammar: grammar, thinkingBudget: budget)), + throwsA( + isA().having( + (error) => error.message, + 'message', + allOf( + contains('thinkingBudgetPauseGrammar'), + contains('v0.1.84+'), + ), + ), + ), + ); + } + expect(fake.completionCalls, 0); + await generate( + const GenerationParams( + grammar: 'root ::= "ok"', + thinkingBudget: budget, + ), + ); + expect(fake.completionOption('thinkingBudget'), paused); + await generate(GenerationParams(grammar: taking)); + expect(fake.completionOption('thinkingBudget'), isNull); + }, + ); +} diff --git a/test/unit/tooling/check_webgpu_bridge_tag_test.dart b/test/unit/tooling/check_webgpu_bridge_tag_test.dart index 551363513..4fb6dc284 100644 --- a/test/unit/tooling/check_webgpu_bridge_tag_test.dart +++ b/test/unit/tooling/check_webgpu_bridge_tag_test.dart @@ -68,37 +68,37 @@ const String _approvedManifestJson = ''' { "artifacts": { "llama_webgpu_bridge.d.ts": { - "sha256": "8a5095c3b5fc62f10612cc6cf3326bc3065529103ad657857f4e1a9fe58dc6ed", - "size_bytes": 22479 + "sha256": "766adcfe0b48c07c26c478a7c9b9dc72bff9386ff297c532baae709ff6367db4", + "size_bytes": 23141 }, "llama_webgpu_bridge.js": { - "sha256": "eaefb36f0d535ac7f76cadecfe97c66c416fc49cb05a4f87144f84499368d79c", - "size_bytes": 316712 + "sha256": "0ecc0ebb7a0f752f48a95de7cefe1e8f2f6b4e955e8a4674f09d49e03dbcb06e", + "size_bytes": 317431 }, "llama_webgpu_bridge_worker.js": { "sha256": "47bbfa0fe897e708455b9497e9aa41d2e94489c329300c33b12936fb3169deca", "size_bytes": 257 }, "llama_webgpu_core.js": { - "sha256": "58b74b403eabfde0703e1f53cdbb9fd34e5eee480bcd0443b32cf97ba48448c0", - "size_bytes": 123402 + "sha256": "f34ad08e95cf6dd1bddd7df31794383ceb6e350553f4487746f54210f8fd9c18", + "size_bytes": 123583 }, "llama_webgpu_core.wasm": { - "sha256": "f20057d88b90de59da5ad35adb8aa49fa4ef5156c9e64e2957d25f8cafe38bdd", - "size_bytes": 11099140 + "sha256": "7e17fac640d50a33a2277036a891b2939f023c4864652b9866236092ac2cb6ee", + "size_bytes": 11099356 }, "llama_webgpu_core_mem64.js": { - "sha256": "282d0934ec2090b13f527e1f01d5740374bdfcc1e75c1f46f9148e7007976a0b", - "size_bytes": 147159 + "sha256": "32b4689944aa4e3a940bddaf31928ab3476e57b9963b918cb38c1281e5688297", + "size_bytes": 147340 }, "llama_webgpu_core_mem64.wasm": { - "sha256": "c2d6548505893b6d89e2a61d6b00a2c502ff03ac8e691c6328c048bbdd4f643a", - "size_bytes": 11390403 + "sha256": "249ab51961555ccf42139b7cb1c05cbd83f62ce3d181aad45671cfc83bb94501", + "size_bytes": 11390643 } }, "assets_repository": "leehack/llama-web-bridge-assets", - "bridge_assets_tag": "v0.1.83", - "bridge_commit": "cb6383fbe7e903764a89f5c80c7f81c219f99769", + "bridge_assets_tag": "v0.1.84", + "bridge_commit": "dc359398bc6445ec15878ddb873555295f957f85", "bridge_repository": "leehack/llama-web-bridge", "capabilities": { "memory64": true, @@ -129,43 +129,43 @@ const String _approvedManifestJson = ''' "emscripten_version": "6.0.8", "files": { "llama_webgpu_bridge.d.ts": { - "sha256": "8a5095c3b5fc62f10612cc6cf3326bc3065529103ad657857f4e1a9fe58dc6ed", - "size_bytes": 22479 + "sha256": "766adcfe0b48c07c26c478a7c9b9dc72bff9386ff297c532baae709ff6367db4", + "size_bytes": 23141 }, "llama_webgpu_bridge.js": { - "sha256": "eaefb36f0d535ac7f76cadecfe97c66c416fc49cb05a4f87144f84499368d79c", - "size_bytes": 316712 + "sha256": "0ecc0ebb7a0f752f48a95de7cefe1e8f2f6b4e955e8a4674f09d49e03dbcb06e", + "size_bytes": 317431 }, "llama_webgpu_bridge_worker.js": { "sha256": "47bbfa0fe897e708455b9497e9aa41d2e94489c329300c33b12936fb3169deca", "size_bytes": 257 }, "llama_webgpu_core.js": { - "sha256": "58b74b403eabfde0703e1f53cdbb9fd34e5eee480bcd0443b32cf97ba48448c0", - "size_bytes": 123402 + "sha256": "f34ad08e95cf6dd1bddd7df31794383ceb6e350553f4487746f54210f8fd9c18", + "size_bytes": 123583 }, "llama_webgpu_core.wasm": { - "sha256": "f20057d88b90de59da5ad35adb8aa49fa4ef5156c9e64e2957d25f8cafe38bdd", - "size_bytes": 11099140 + "sha256": "7e17fac640d50a33a2277036a891b2939f023c4864652b9866236092ac2cb6ee", + "size_bytes": 11099356 }, "llama_webgpu_core_mem64.js": { - "sha256": "282d0934ec2090b13f527e1f01d5740374bdfcc1e75c1f46f9148e7007976a0b", - "size_bytes": 147159 + "sha256": "32b4689944aa4e3a940bddaf31928ab3476e57b9963b918cb38c1281e5688297", + "size_bytes": 147340 }, "llama_webgpu_core_mem64.wasm": { - "sha256": "c2d6548505893b6d89e2a61d6b00a2c502ff03ac8e691c6328c048bbdd4f643a", - "size_bytes": 11390403 + "sha256": "249ab51961555ccf42139b7cb1c05cbd83f62ce3d181aad45671cfc83bb94501", + "size_bytes": 11390643 } }, - "github_run_id": "42479331139", - "github_run_url": "https://github.com/leehack/llama-web-bridge/actions/runs/42479331139", + "github_run_id": "43272135497", + "github_run_url": "https://github.com/leehack/llama-web-bridge/actions/runs/43272135497", "llama_cpp_commit": "cd959bae99ff4bed22bef2604f45241c09bbb8c8", "llama_cpp_tag": "v0.9.0", "native_commit": "a604639131105c42a25a3e7384ca10314d385f80", "native_manifest_sha256": "0d2119cdcf34d347a96a996423094498de3393026efb6ad1a96ba286b621c39e", "native_release_tag": "v0.9.0", "native_repository": "leehack/llamadart-native", - "orchestrator_correlation_id": "manual-v0.1.83", + "orchestrator_correlation_id": "manual-v0.1.84", "qualification_gates": { "multimodal": "passed", "speech_to_text": "required-automated-qualification", @@ -174,9 +174,9 @@ const String _approvedManifestJson = ''' }, "release_channel": "stable", "release_rebuild": 0, - "release_tag": "v0.1.83", + "release_tag": "v0.1.84", "schema_version": 2, - "source_commit": "cb6383fbe7e903764a89f5c80c7f81c219f99769", + "source_commit": "dc359398bc6445ec15878ddb873555295f957f85", "source_repository": "leehack/llama-web-bridge", "unproven_capabilities": { "hardware_gpu_acceleration": "unavailable-on-hosted-runners", @@ -197,7 +197,7 @@ Future> _verifyManifestJson( }) { final bytes = utf8.encode(manifestJson); return verifyManifest( - expectedTag: 'v0.1.83', + expectedTag: 'v0.1.84', expectedLlamaCppTag: bridgeLlamaCppTag, expectedLlamaCppCommit: bridgeLlamaCppCommit, expectedBridgeCommit: bridgeSourceCommit, @@ -223,7 +223,7 @@ String _currentReleaseNotes(String path) { void main() { test('default assets include the qualified media embedding API', () { - expect(readPinnedBridgeTag(Directory.current), 'v0.1.83'); + expect(readPinnedBridgeTag(Directory.current), 'v0.1.84'); expect(bridgeNativeReleaseTag, 'v0.9.0'); expect(bridgeApprovedNativePin, 'v0.9.0'); expect( @@ -285,7 +285,7 @@ void main() { 'website/docs/changelog/recent-releases.md', ]) { final current = _currentReleaseNotes(path); - expect(current, contains('`v0.1.83`'), reason: path); + expect(current, contains('`v0.1.84`'), reason: path); expect(current, contains(bridgeManifestSha256), reason: path); expect( current, @@ -1074,8 +1074,8 @@ void main() { }); test('immutable release identity stays pinned to the approved release', () { - expect(bridgeAssetsReleaseId, '438242670'); - expect(bridgeAssetsTagCommit, '44aa84a8a51701dc9c4f82c21e6ee3e774919c20'); + expect(bridgeAssetsReleaseId, '443341300'); + expect(bridgeAssetsTagCommit, '0a3c45f24fd6293050e1772bed4a425b2b5a8493'); }); test('accepts equivalent CRLF documentation passages', () { diff --git a/test/unit/tooling/test_matrix_test.dart b/test/unit/tooling/test_matrix_test.dart index 9bcd22401..e2e13f34c 100644 --- a/test/unit/tooling/test_matrix_test.dart +++ b/test/unit/tooling/test_matrix_test.dart @@ -352,6 +352,7 @@ void main() { 'local-only', 'test/e2e/webgpu/lazy_grammar_e2e_test.dart', 'test/e2e/webgpu/prompt_penalty_grammar_e2e_test.dart', + 'test/e2e/webgpu/thinking_budget_grammar_e2e_test.dart', ]); for (final step in steps) { expect(step.workingDirectory, '/repo'); diff --git a/tool/ci/test_shard_weights.json b/tool/ci/test_shard_weights.json index d006e987d..b9d9b79a4 100644 --- a/tool/ci/test_shard_weights.json +++ b/tool/ci/test_shard_weights.json @@ -53,6 +53,7 @@ "test/e2e/webgpu/multimodal_regression_e2e_test.dart": 0, "test/e2e/webgpu/prompt_penalty_grammar_e2e_test.dart": 0, "test/e2e/webgpu/speech_task_cancellation_e2e_test.dart": 0, + "test/e2e/webgpu/thinking_budget_grammar_e2e_test.dart": 0, "test/integration/apple_companion_flutter_cache_test.dart": 0, "test/integration/backends/litert_lm/engine_create_failure_test.dart": 3990, "test/integration/backends/litert_lm/qwen_content_contract_test.dart": 2973, @@ -255,6 +256,7 @@ "test/unit/backends/webgpu/webgpu_lora_alias_test.dart": 0, "test/unit/backends/webgpu/webgpu_lora_test.dart": 0, "test/unit/backends/webgpu/webgpu_speculative_test.dart": 0, + "test/unit/backends/webgpu/webgpu_thinking_budget_grammar_test.dart": 0, "test/unit/backends/webgpu/webgpu_url_test.dart": 0, "test/unit/backends/windows_runtime_libraries_test.dart": 184, "test/unit/backends/worker_log_message_test.dart": 181, @@ -654,6 +656,7 @@ "test/e2e/webgpu/multimodal_regression_e2e_test.dart": 0, "test/e2e/webgpu/prompt_penalty_grammar_e2e_test.dart": 0, "test/e2e/webgpu/speech_task_cancellation_e2e_test.dart": 0, + "test/e2e/webgpu/thinking_budget_grammar_e2e_test.dart": 0, "test/integration/apple_companion_flutter_cache_test.dart": 0, "test/integration/backends/litert_lm/engine_create_failure_test.dart": 0, "test/integration/backends/litert_lm/qwen_content_contract_test.dart": 16095, @@ -856,6 +859,7 @@ "test/unit/backends/webgpu/webgpu_lora_alias_test.dart": 6997, "test/unit/backends/webgpu/webgpu_lora_test.dart": 6418, "test/unit/backends/webgpu/webgpu_speculative_test.dart": 6129, + "test/unit/backends/webgpu/webgpu_thinking_budget_grammar_test.dart": 0, "test/unit/backends/webgpu/webgpu_url_test.dart": 5010, "test/unit/backends/windows_runtime_libraries_test.dart": 0, "test/unit/backends/worker_log_message_test.dart": 0, @@ -1255,6 +1259,7 @@ "test/e2e/webgpu/multimodal_regression_e2e_test.dart": 0, "test/e2e/webgpu/prompt_penalty_grammar_e2e_test.dart": 0, "test/e2e/webgpu/speech_task_cancellation_e2e_test.dart": 0, + "test/e2e/webgpu/thinking_budget_grammar_e2e_test.dart": 0, "test/integration/apple_companion_flutter_cache_test.dart": 0, "test/integration/backends/litert_lm/engine_create_failure_test.dart": 3929, "test/integration/backends/litert_lm/qwen_content_contract_test.dart": 1599, @@ -1457,6 +1462,7 @@ "test/unit/backends/webgpu/webgpu_lora_alias_test.dart": 0, "test/unit/backends/webgpu/webgpu_lora_test.dart": 0, "test/unit/backends/webgpu/webgpu_speculative_test.dart": 0, + "test/unit/backends/webgpu/webgpu_thinking_budget_grammar_test.dart": 0, "test/unit/backends/webgpu/webgpu_url_test.dart": 0, "test/unit/backends/windows_runtime_libraries_test.dart": 330, "test/unit/backends/worker_log_message_test.dart": 200, @@ -1856,6 +1862,7 @@ "test/e2e/webgpu/multimodal_regression_e2e_test.dart": 0, "test/e2e/webgpu/prompt_penalty_grammar_e2e_test.dart": 0, "test/e2e/webgpu/speech_task_cancellation_e2e_test.dart": 0, + "test/e2e/webgpu/thinking_budget_grammar_e2e_test.dart": 0, "test/integration/apple_companion_flutter_cache_test.dart": 0, "test/integration/backends/litert_lm/engine_create_failure_test.dart": 5283, "test/integration/backends/litert_lm/qwen_content_contract_test.dart": 603, @@ -2058,6 +2065,7 @@ "test/unit/backends/webgpu/webgpu_lora_alias_test.dart": 0, "test/unit/backends/webgpu/webgpu_lora_test.dart": 0, "test/unit/backends/webgpu/webgpu_speculative_test.dart": 0, + "test/unit/backends/webgpu/webgpu_thinking_budget_grammar_test.dart": 0, "test/unit/backends/webgpu/webgpu_url_test.dart": 0, "test/unit/backends/windows_runtime_libraries_test.dart": 238, "test/unit/backends/worker_log_message_test.dart": 273, diff --git a/tool/testing/check_webgpu_bridge_tag.dart b/tool/testing/check_webgpu_bridge_tag.dart index f9fd8e70f..110771649 100644 --- a/tool/testing/check_webgpu_bridge_tag.dart +++ b/tool/testing/check_webgpu_bridge_tag.dart @@ -372,7 +372,7 @@ const String bridgeLlamaCppTag = 'v0.9.0'; const String bridgeLlamaCppCommit = 'cd959bae99ff4bed22bef2604f45241c09bbb8c8'; /// The exact bridge source commit used to build the pinned bridge assets. -const String bridgeSourceCommit = 'cb6383fbe7e903764a89f5c80c7f81c219f99769'; +const String bridgeSourceCommit = 'dc359398bc6445ec15878ddb873555295f957f85'; /// Canonical repository identities recorded in the approved manifest. const String bridgeAssetsRepository = 'leehack/llama-web-bridge-assets'; @@ -393,10 +393,10 @@ const String bridgeApprovedNativePin = 'v0.9.0'; const String bridgeApprovedNativeUpstreamAhead = 'v0.9.0'; /// The asset repository release that published the pinned bridge assets. -const String bridgeAssetsReleaseId = '438242670'; +const String bridgeAssetsReleaseId = '443341300'; /// The asset repository commit the pinned bridge asset tag points at. -const String bridgeAssetsTagCommit = '44aa84a8a51701dc9c4f82c21e6ee3e774919c20'; +const String bridgeAssetsTagCommit = '0a3c45f24fd6293050e1772bed4a425b2b5a8493'; /// SHA-256 hash of the exact approved published manifest.json. /// diff --git a/tool/testing/run_local_e2e.dart b/tool/testing/run_local_e2e.dart index 43104e421..f073ef4bc 100644 --- a/tool/testing/run_local_e2e.dart +++ b/tool/testing/run_local_e2e.dart @@ -1394,8 +1394,9 @@ List buildLocalE2eScenarios({String? projectRoot}) { name: 'web-lazy-grammar', group: LocalE2eScenarioGroup.webSmoke, description: - 'Check pinned-asset lazy grammar activation, custom roots and ' - 'typed rejection/reuse in direct and worker CPU Wasm runtimes.', + 'Check pinned-asset lazy grammar activation, custom roots, ' + 'thinking-budget grammar and typed rejection/reuse in direct and ' + 'worker CPU Wasm runtimes.', requiresDevice: false, stepsBuilder: (context) => [ LocalE2eCommandStep( @@ -1420,8 +1421,11 @@ List buildLocalE2eScenarios({String? projectRoot}) { 'local-only', 'test/e2e/webgpu/lazy_grammar_e2e_test.dart', 'test/e2e/webgpu/prompt_penalty_grammar_e2e_test.dart', + 'test/e2e/webgpu/thinking_budget_grammar_e2e_test.dart', ], - description: 'Web lazy grammar, prompt penalties and error recovery', + description: + 'Web lazy grammar, prompt penalties, thinking-budget grammar ' + 'and error recovery', ), ], ), diff --git a/tool/testing/test_matrix.dart b/tool/testing/test_matrix.dart index aebad30f2..c490b01e0 100644 --- a/tool/testing/test_matrix.dart +++ b/tool/testing/test_matrix.dart @@ -878,7 +878,9 @@ const List testMatrixRows = [ 'lazy word/pattern/pattern-full/token activation after visible text, ' 'custom roots, grammar prompt penalties and typed rejection/reuse through ' 'pinned assets in direct and worker CPU Wasm runtimes; prompt penalties ' - 'cover wasm32 and memory64; no tool accuracy or GPU claim', + 'cover wasm32 and memory64; a Qwen3.5 response-format grammar stays ' + 'applied through a budgeted thought, and assets without ' + 'thinkingBudgetPauseGrammar reject it; no tool accuracy or GPU claim', command: 'dart run tool/testing/run_local_e2e.dart --scenario web-lazy-grammar', useWhen: diff --git a/website/docs/changelog/recent-releases.md b/website/docs/changelog/recent-releases.md index 3e71e0827..40239f351 100644 --- a/website/docs/changelog/recent-releases.md +++ b/website/docs/changelog/recent-releases.md @@ -15,6 +15,7 @@ validate against the exact release tag notes. - Fixed constrained JSON (`responseFormat`) ignoring the grammar after a `ThinkingBudget` closes a template-opened thought on native llama.cpp (Qwen3.5 and similar), and the same for required tool calls on templates whose grammar takes that thought. - Fixed a required tool call on Seed-OSS being unconstrained after a `ThinkingBudget` closes a thought the model opened, on native llama.cpp ([#1958](https://github.com/leehack/llamadart/issues/1958)). +- Aligned default WebGPU bridge assets to `v0.1.84`: constrained JSON (`responseFormat`) and Seed-OSS required tool calls stay constrained after a `ThinkingBudget` closes a thought on WebGPU, and older assets throw `LlamaUnsupportedException` for that combination ([#1954](https://github.com/leehack/llamadart/issues/1954)). Web keeps native llama.cpp `v0.9.0` parity (upstream `v0.9.0@cd959bae99ff4bed22bef2604f45241c09bbb8c8`) and `@litert-lm/core@0.19.0`. Manifest SHA-256: `55abeb377597148156059a8ee5400f2314c6254ad91b57db65b709e304c0dea0`. ## 0.19.7 diff --git a/website/docs/platforms/webgpu-bridge.md b/website/docs/platforms/webgpu-bridge.md index 95305a85b..8cc8d5c07 100644 --- a/website/docs/platforms/webgpu-bridge.md +++ b/website/docs/platforms/webgpu-bridge.md @@ -287,7 +287,12 @@ The feature-by-runtime table is in the probe reports `penaltyPromptTokens`), as native llama.cpp does. `ToolChoice.auto` requests that skip the lazy tool-call grammar keep the prompt out, as native does under that grammar ([llama-web-bridge#187](https://github.com/leehack/llama-web-bridge/pull/187)). - A thinking budget is text-only. A LoRA adapter is a `ModelSource` whose URL the bridge downloads once + A thinking budget is text-only. As on native llama.cpp, a thinking budget keeps a + grammar applied through the thought when that grammar takes it (constrained JSON + on Qwen3.5, required tool calls on Seed-OSS) and pauses every other grammar until + reasoning ends; the former needs assets whose probe reports + `thinkingBudgetPauseGrammar` (`v0.1.84+`) and throws `LlamaUnsupportedException` + without it. A LoRA adapter is a `ModelSource` whose URL the bridge downloads once per model load (a `ModelSource.path` is a URL relative to the document, or a `blob:` URL), through Cache Storage unless the URL carries credentials; reload adapters after loading another model. An aLoRA adapter throws `LlamaUnsupportedException`, and an adapter the bridge cannot load, such as @@ -401,10 +406,10 @@ first model load, for diagnosis or controlled deployments: ## Pinned bridge assets -This llamadart version pins bridge assets `v0.1.83`, which include the +This llamadart version pins bridge assets `v0.1.84`, which include the API-version-1 media embedding contract. -- The pinned `v0.1.83` bridge assets embed llama.cpp `v0.9.0`, matching the native runtime. +- The pinned `v0.1.84` bridge assets embed llama.cpp `v0.9.0`, matching the native runtime. Provenance and the pinned manifest digest are in [`doc/webgpu_bridge.md`](https://github.com/leehack/llamadart/blob/main/doc/webgpu_bridge.md). @@ -412,7 +417,7 @@ Provenance and the pinned manifest digest are in In a llamadart checkout, vendor the pinned assets into the chat app with: ```bash -WEBGPU_BRIDGE_ASSETS_TAG=v0.1.83 ./scripts/fetch_webgpu_bridge_assets.sh +WEBGPU_BRIDGE_ASSETS_TAG=v0.1.84 ./scripts/fetch_webgpu_bridge_assets.sh ``` The chat app bootstrap takes its CDN source from these globals: @@ -420,7 +425,7 @@ The chat app bootstrap takes its CDN source from these globals: ```html ``` @@ -459,3 +464,4 @@ probe, not the tag. | `v0.1.75` | A draft context that does not fit throws `LlamaUnsupportedException` and the target stays usable | The draft leaks until the runtime aborts | | `v0.1.76` | The projector copy behind the repeated wasm32 media batch abort is freed (wasm32 stays unqualified) | Repeated media batches can abort on wasm32 | | `v0.1.79+` | A worker image or audio `embed` gets 10 minutes per input | Two minutes per request | +| `v0.1.84+` | A thinking budget keeps a grammar that takes the thought applied (`thinkingBudgetPauseGrammar`) | That combination throws `LlamaUnsupportedException` | From 65c62f433b766d6afc08c80531899a4bdae6b9f8 Mon Sep 17 00:00:00 2001 From: leehack Date: Thu, 12 Nov 2026 13:54:34 -0500 Subject: [PATCH 2/2] Reconcile the thinkingBudget WebGPU Dartdoc after merging main (#1954) --- lib/src/core/models/inference/generation_params.dart | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lib/src/core/models/inference/generation_params.dart b/lib/src/core/models/inference/generation_params.dart index 429460d39..5bfe3326c 100644 --- a/lib/src/core/models/inference/generation_params.dart +++ b/lib/src/core/models/inference/generation_params.dart @@ -956,13 +956,13 @@ class GenerationParams { /// models the thought the model opens (Seed-OSS's required-call root, /// starting with `seed-thought`) applied while budgeted reasoning is /// active. It pauses every other grammar, lazy grammars included, until - /// reasoning has ended. WebGPU does the same on bridge assets that report - /// `thinkingBudgetPauseGrammar` (v0.1.84+) and rejects the keep-applied - /// grammars on older assets. Native LiteRT-LM applies it to chat generation on + /// reasoning has ended. Native LiteRT-LM applies it to chat generation on /// bundles with a thinking channel, which it closes with the bundle's own /// end marker; it rejects a zero budget and [ThinkingBudget.forcedMessage]. - /// WebGPU bridges that report `thinkingBudget` support it and pause every - /// grammar during reasoning. + /// WebGPU bridges that report `thinkingBudget` support it and pause grammars + /// the same way as native llama.cpp; keeping a grammar applied through the + /// thought needs assets that report `thinkingBudgetPauseGrammar` (v0.1.84+), + /// and older assets reject that combination. /// Other backends reject this setting rather than silently ignoring it. final ThinkingBudget? thinkingBudget;