Repository navigation
fix(qwen3.5): capture logprobs on the prefill draw when the think block is prefilled - #780
Open
manueldomke wants to merge 6 commits into
Open
manueldomke wants to merge 6 commits into
manueldomke wants to merge 6 commits into
Conversation
Qwen3_5VL::insert() and Qwen3_6_MoE::insert() call _shared_insert(), which
prefills and draws one token into `last_token`. Their generate() then
force-feeds the think block and samples again; the drawn token is never used,
and the only reference to it is an assert.
sample() records every drawn token in the repetition and frequency windows
first. So a token that was thrown away counts as generated text. For a model
that answers without a thinking block, the token predicted after `assistant`
IS the answer letter -- it is pushed down by count * freq_penalty +
pre_penalty at exactly the position where it is needed, which for a single
occurrence is 1.0 + 1.5 = 2.5 logits (Qwen3.5-VL's defaults; its rep_penalty
is 1.0 and does nothing).
Measured on Qwen3.5-2B, temperature 0, default penalties, before and after,
nothing else changed:
prompt Q before B after A
prompt R before B after A
A is the argmax of the unpenalised distribution (0.339 against 0.299 for B),
so the answer is no longer the penalised one.
Sampler::sample() and sample_greedy() take a record_in_history flag,
defaulting to true, so no other model changes behaviour. sample() forwards it
to sample_greedy() on the top_k == 1 path, and _shared_insert() passes it
through -- both paths have to honour it, or the same prompt answers
differently depending on a sampling knob:
top_k=1 B (greedy path)
top_k=40 A (sampled path) before both flags were in place
Co-Authored-By: DeepSeek V4 Flash (Default Thinking)
/v1/chat/completions and /v1/completions wrote a hard-coded
`{"logprobs", nullptr}` into their responses. This fills the field from the
sampler, in OpenAI's shape, and only when the request asks for it.
For a model that answers with a single token -- a classifier, a router, a
decision model -- the letter alone says nothing about how sure the model was.
sampler.hpp/.cpp capture_if_requested(), log_sum_exp_all() and a table of
exp() for every bf16 value; the sampling paths capture
once per draw while a request has the sampler armed
automodel.hpp/.cpp top_logprob_t, logprobs_entry_t,
_request_logprobs_capture() and _record_top_logprobs()
modeling_qwen3_5vl records where a token reaches the caller's text, not
where it was drawn
rest_handler.cpp reads the request, builds both response shapes
One entry per token of message.content, in the same order, so a client that
pairs entries with tokens stays aligned. Recording happens where the token is
written to the response, which is what keeps the two in step: a token the
engine filters out gets no entry, and neither does EOS. A 17-token answer
comes back with 16 entries. An answer that carries a tool call gets null,
because message.content is empty and there are no tokens to describe.
logprob is a real log-probability -- logit minus logsumexp over the WHOLE
vocabulary, after the penalties and before the temperature. It is NOT
renormalised over the entries listed, so those do not sum to 1; the rest of
the vocabulary lies outside the list.
The logsumexp is exact and cheap: the logits arrive as bf16 from the engine,
so there are only 65536 possible values, and a table of exp() for each of them
(512 KB, fits in L2) replaces a quarter of a million exp() calls with as many
reads. Values the penalties moved off the bf16 grid take the exact call. The
result differs from the shifted computation by about one float ulp, which is
rounding, not approximation. For the alternatives, N <= top_k needs no sort at
all: top_k_logits is already sorted by logit with the same tie-break, so its
beginning is the top-n.
Cost, wall clock on Qwen3.5-2B, 300 tokens, nothing else running,
platform_profile=balanced. The runs drift by about 2 % on their own, so this
is an upper bound rather than a figure:
without logprobs 10.55 10.46 10.35 s
with logprobs 10.57 s at most about 1 %
The sampler is disarmed at the start of every request, before anything that
can return early; it is armed only in Qwen3.5-VL's generate(), and only when
the request asks for logprobs.
The two request shapes differ, as they do upstream: chat takes `logprobs` as a
boolean plus `top_logprobs` (0-20), completions takes `logprobs` as an integer
0-5 and has no `top_logprobs`. For completions 0 is a valid request: it asks
for the drawn token's own logprob with no alternatives.
With a decoding constraint (response_format, tool_choice) the capture sees
the masked logits, the ones the draw used, so logprob is normalised over the
tokens the constraint allows. Alternatives the constraint forbids sit at -inf
and are not reported, since -inf has no JSON number.
Limitations, unchanged by this patch:
- only Qwen3.5-VL reports; every other model still returns null
- streaming responses carry no logprobs
- with think:true the answer gets null: the tokens of the reasoning block
land in reasoning_content, not in content
Seen while working on this, not touched here: the AVX2 argmax in
sample_greedy() breaks ties within a lane by the lowest index but merges lanes
by the lowest lane, so an exact tie at the top can pick a higher id than the
scalar path would.
Co-Authored-By: DeepSeek V4 Flash (Default Thinking)
Qwen3_5VL::generate() draws the first token before its decode loop and checked the limit only at the bottom of the loop, after the next draw. So every request that ran into max_tokens got one token more than it asked for, and paid one more forward() for it: max_tokens=1 returned "A\n" with completion_tokens 2. The loop now checks the limit first, before it draws. Its shape follows _shared_generate(): the context limit is checked inside the loop, and reaching max_tokens still wins over it, so finish_reason is unchanged. Measured on Qwen3.5-2B (NPU2), 20 independent one-token prompts, three alternating rounds against main: median wall clock 604 -> 574 ms per request, completion_tokens 2 -> 1. The first token of every answer and two 60-token generations are identical to main. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Without thinking, the chat template ends with an empty think block. Qwen3_5VL::insert() cut those four tokens off before the prefill, so the checkpoint lands where the next turn of a conversation diverges, and generate() then fed them in with one forward() each. Only after those four decode steps came the draw for the answer's first token. A single-turn model never returns to that checkpoint. For such a model the block is now prefilled with the prompt, and the token _shared_insert() draws from the prefill logits is the answer's first token. Multi-turn models keep the old path, because there the checkpoint pays off. So does thinking, where the model writes the block itself. Under a decoding constraint the draw is kept as well. The constraint starts from the prompt tail, which now ends behind the closed block, and it has already accepted the token; nothing is rolled back. The catalog's "single-turn" label now also sets AutoModel::single_turn. Until now only three classes set the flag themselves, all with the label, so nothing changes for them. It is only ever switched on, never off. Measured on Qwen3.5-2B (NPU2) with the label set, 20 independent one-token prompts, three alternating rounds against main: median wall clock 604 -> 486 ms per request. Prefill time is unchanged (430 vs 431 ms); the four tokens it gains cost nothing measurable. The numbers are not bit-identical. The template's tokens are exactly the four ids generate() used to feed ([248068, 271, 248069, 271]), but they now go through the prefill kernels instead of four decode steps. Against main with the discarded-sample fix (ROCm#776) 1 of 20 answers differs, and that was an exact tie (A and C both at logprob -1.939). One of two 60-token generations diverges after 90 characters. With logprobs applied on top, the log-odds between options move by 0.25 nats median, 0.5 at most. Co-Authored-By: Claude Opus 5.5 <[email protected]>
_shared_insert() skips the prefix a request shares with token_history. If the request matches the history completely, nothing is left to prefill, _chunked_insert() returns an empty buffer, and Sampler::sample() reads it: a segfault at address 0. With the empty think block prefilled, a single-turn Qwen3.5 request puts exactly its own tokens into token_history. The first generated token is never added there. On its own this change cannot hit the bug: with max_tokens=1 the decode loop still draws one more token and appends it. Together with the max_tokens fix from the companion PR the loop no longer runs, and the same request sent twice crashed the server on the second one. Reproduced under gdb: the crashing request logs no "Prefill chunk" line. A complete match is now treated like a divergence: clear and prefill everything. Stepping back by one token is not an option, because the linear attention layers carry recurrent state that cannot be rewound. With both changes applied, eight identical requests in a row pass, and answers and confidences are unchanged. The extra prefill is not measurable, because a single-turn model cleared its context on these requests before anyway. Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ck is prefilled Builds on ROCm#777 (logprobs, which contains ROCm#776) and on the single-turn prefill change. With the empty think block prefilled, the token _shared_insert() draws is the answer's first token. It must be counted in the repetition window, and a logprobs request must capture it. Until now generate() armed the capture only on the old path, before the draw that follows the forced block, so a single-turn model returned no logprobs at all. _shared_insert() takes capture_logprobs and arms the sampler right before its own draw, after the disarming at its top. Qwen3_5VL::insert() passes it, together with record_sampled_token, exactly when the block was prefilled. Measured on Qwen3.5-2B (NPU2), the same binary with and without the "single-turn" label, 20 one-token prompts: 572 -> 455 ms median per request. Both report one logprobs entry per answer token, the chosen entry matches the content, and both return null without logprobs and with think:true. Co-Authored-By: Claude Opus 5.5 <[email protected]>
manueldomke
force-pushed
the
decider-integration
branch
from
October 21, 2026 11:51
2b0692d to
1d22326
Compare
Author
1 task done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
This PR brings #777 (logprobs, which contains #776) together with
#778 and #779.
With the think block prefilled, the token
_shared_insert()draws is theanswer's first token. #777 arms the capture only in
generate(), on the oldpath, and
_shared_insert()disarms at its top. So a single-turn modelreturned no logprobs at all.
Technical Details
New here is one commit,
fix(qwen3.5): capture logprobs on the prefill draw when the think block is prefilled._shared_insert()takescapture_logprobsand arms the sampler right before its own draw, after thedisarming at its top.
Qwen3_5VL::insert()passes it, together withrecord_sampled_token, exactly when the block was prefilled. A thrown-awaydraw is never captured.
The other commits are #776, #777, #778 and #779 including
its guard, cherry-picked. Once those are merged this reduces to the one commit,
and I'll rebase.
Test Plan
single-turnlabel, so only theprefill path differs. 20 one-token prompts,
temperature: 0,logprobs: true, top_logprobs: 3.think: true.Test Result
Against
main(604 ms) that is −149 ms, about 25 %. Answers differ in 1 of 20prompts, the exact tie from #779. The log-odds between options shift
by 0.25 nats median, 0.5 at most.
Real workload. A decision game made 170 decisions against this build, many
of them identical requests in a row. There was no crash, and in-game latency
went from 936 to 774 ms compared with #776 + #777 alone. No accuracy change is
claimed here: answers and latency changed together, and the game is
time-sensitive.
Submission Checklist
🤖 Generated with Claude Code