From e4ad765e3f654c3da3f9be604bf767b0c51cf221 Mon Sep 17 00:00:00 2001 From: leehack Date: Thu, 12 Nov 2026 15:09:28 -0500 Subject: [PATCH] Document the end-tag-rejecting grammar failure and the Qwen3.5-only smoke repro (#213 follow-up) --- CONTRIBUTING.md | 3 ++- docs/api.md | 2 +- scripts/smoke/grammar.mjs | 5 ++++- src/core/generation.inc | 6 ++++-- 4 files changed, 11 insertions(+), 5 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b1e55a4..8f567b4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -241,7 +241,8 @@ CI runs it twice: with the state-persistence model, then with the multimodal model. Pass each model's `--model-url` (or `--model-path`) and `--model-sha256` pin from the commands above; the smoke defaults to the `LLAMA_WEBGPU_SMOKE_MODEL_URL` and `LLAMA_WEBGPU_SMOKE_MODEL_SHA256` environment -variables: +variables. Its end-tag-rejecting grammar check reproduces the original bug +only with the multimodal (Qwen3.5) model; the tiny model passes it either way: ```bash node scripts/smoke/grammar.mjs \ diff --git a/docs/api.md b/docs/api.md index 7182c51..879666f 100644 --- a/docs/api.md +++ b/docs/api.md @@ -326,7 +326,7 @@ Common `options` keys: | `grammarLazy` | `true` applies `grammar` only after a `grammarTriggers` entry fires, as llama.cpp's lazy grammar (`llama-server`'s `grammar_lazy`) does; see [Lazy grammars](#lazy-grammars). Needs a non-empty `grammar` and at least one trigger. Omitted or `null` means `false`. | | `grammarTriggers` | Array of `{ type: 'word' \| 'pattern' \| 'pattern-full', value }` or `{ type: 'token', token }` that fire a lazy grammar. Always validated; applied only when `grammarLazy` is `true`. | | `seed` | Integer seed; random when omitted. | -| `thinkingBudget` | `{ maxTokens, startTag, endTag, forcedMessage?, pauseGrammar? }`: llama.cpp's reasoning budget. After `maxTokens` tokens inside a block opened by `startTag`, plus any that finish a UTF-8 character, the sampler forces `forcedMessage` (default empty) and `endTag`. A block the prompt leaves open counts from the start of generation, so `maxTokens: 0` forces the end at once. Each new `startTag` gets a fresh budget. A grammar constrains `startTag` like any other text, then pauses until `endTag` completes, unless `pauseGrammar` is `false`: that keeps the grammar applied inside the block, for one that takes the block and `endTag` itself (such as one written for a prompt that ends inside the block; paused, it resumes at its start and takes the reply as reasoning) and so accepts `forcedMessage` and `endTag` wherever the budget can run out. Omitted, `null` or `true` pauses it; `false` needs the `thinkingBudgetPauseGrammar` capability. `maxTokens` is an integer from `0` to `2147483647`; the tags must be non-empty. Each tag is tokenized on its own, as native llamadart does: with a vocabulary that adds a leading space to text, such as Llama 2's SentencePiece, a tag in the prompt matches only after a space, and the forced end tag starts with one. Text-only prompts: with `parts` it rejects. | +| `thinkingBudget` | `{ maxTokens, startTag, endTag, forcedMessage?, pauseGrammar? }`: llama.cpp's reasoning budget. After `maxTokens` tokens inside a block opened by `startTag`, plus any that finish a UTF-8 character, the sampler forces `forcedMessage` (default empty) and `endTag`. A block the prompt leaves open counts from the start of generation, so `maxTokens: 0` forces the end at once. Each new `startTag` gets a fresh budget. A grammar constrains `startTag` like any other text, then pauses until `endTag` completes, unless `pauseGrammar` is `false`: that keeps the grammar applied inside the block, for one that takes the block and `endTag` itself (such as one written for a prompt that ends inside the block; paused, it resumes at its start and takes the reply as reasoning) and so accepts `forcedMessage` and `endTag` wherever the budget can run out; a kept grammar that rejects the forced end tag fails the generation with `Grammar rejected every candidate token`. Omitted, `null` or `true` pauses it; `false` needs the `thinkingBudgetPauseGrammar` capability. `maxTokens` is an integer from `0` to `2147483647`; the tags must be non-empty. Each tag is tokenized on its own, as native llamadart does: with a vocabulary that adds a leading space to text, such as Llama 2's SentencePiece, a tag in the prompt matches only after a space, and the forced end tag starts with one. Text-only prompts: with `parts` it rejects. | | `onUsage(usage)` | Called at most once, with the `CompletionUsage` of the last attempt whose generation returned, before the promise resolves or rejects with an `AbortError`. Not called when no generation returned: other rejections, a skipped multimodal warmup, or a worker failure after a cancel. | | `onToken(piece, currentText)` | Token callback. By default `piece` is a `Uint8Array` containing complete UTF-8 characters; see [Streamed text](#streamed-text). `currentText` is the text streamed so far. Direct runtime mode provides it by default; worker mode provides `''` unless `emitCurrentTextOnToken: true` is set. | | `signal` | `AbortSignal`; aborting cancels this operation and rejects with `AbortError`. | diff --git a/scripts/smoke/grammar.mjs b/scripts/smoke/grammar.mjs index c2f16db..8f26a0a 100644 --- a/scripts/smoke/grammar.mjs +++ b/scripts/smoke/grammar.mjs @@ -51,7 +51,10 @@ With a thinking budget it checks that a grammar pauses inside the reasoning block by default; that \`\`pauseGrammar: false\`\` keeps a grammar that takes the block and its end tag itself applied through the forced end tag; and that a grammar rejecting the forced end tag fails with the grammar-rejection error -instead of finishing the block without it. +instead of finishing the block without it. Only the Qwen3.5 run reproduces the +original bug (the end-tag-rejecting grammar finishing the block without its end +tag); the tiny-model run passes with or without the fix, so CI must keep the +Qwen3.5 run. It then checks \`\`minP\`\` and \`\`presencePenalty\`\` with a fixed seed. Every compared completion reuses the same cached prompt, so each run starts from the diff --git a/src/core/generation.inc b/src/core/generation.inc index c48f45a..4e66441 100644 --- a/src/core/generation.inc +++ b/src/core/generation.inc @@ -135,8 +135,10 @@ llama_token sample_next_token(const int32_t batch_index = -1) { token = cur_p.data[cur_p.selected].id; // Every candidate is rejected only when the grammar can't be continued // with this vocabulary; accepting the token would abort. A -inf pick means - // the grammar and the reasoning budget's forced end tag excluded each other, - // and the budget would otherwise finish the block without its end tag. + // the sampler left no finite candidate: the grammar and the reasoning + // budget's forced end tag excluded each other, or a subnormal temp with no + // budget overflowed the scaled logits. Either way fail closed instead of + // emitting a token the grammar never allowed. if (cur_p.data[cur_p.selected].logit == -INFINITY || !grammar_accepts_token(token)) { set_error("Grammar rejected every candidate token");