[manueldomke]

Motivation

Qwen3_5VL::insert() and Qwen3_6_MoE::insert() call _shared_insert(), which
prefills and draws one token into last_token. Their generate() then
force-feeds the think block and samples again -- the drawn token is never used,
and the only reference to it is an assert.

sample() records every drawn token in the repetition and frequency windows
first. A token that was thrown away therefore counts as generated text. For a
model that answers without a thinking block the token predicted after
assistant IS the answer letter, so it is pushed down by
count * freq_penalty + pre_penalty at exactly the position where it is
needed -- 1.0 + 1.5 = 2.5 logits with this model's defaults.

The visible effect is that the model answers against its own preference. On
Write one sentence about the sea. it says A vast, ever-changing blue expanse... where it says The sea is a vast, ever-changing blue mirror...
once the discarded sample is no longer counted. Both are fine sentences; the
second is the one the model chose.

Three Qwen3.5 fine-tunes converted for the NPU are affected directly: they
answer with a single letter, and that letter is the one being pushed down.

• seimanu/decider-0.8b-NPU2
• seimanu/decider-2b-NPU2
• seimanu/decider-4b-NPU2

Technical Details

Sampler::sample() and sample_greedy() take a record_in_history flag,
defaulting to true, so no other model changes behaviour. sample() forwards
it to sample_greedy() on the top_k == 1 path, and _shared_insert() passes
it through from its caller. Qwen3_5VL::insert() and Qwen3_6_MoE::insert()
pass false for the draw they discard.

Both paths have to honour the flag. With only one of them in place the same
prompt answers differently depending on a sampling knob:

top_k=1    B     (greedy path)
top_k=40   A     (sampled path)

Scope, and what it does not claim. Two models are covered, because they are
the two where the draw is provably discarded: in both, last_token appears
only in an assert and is read nowhere. The flag defaults to true, so every
other model keeps its behaviour -- which also means a model with the same
pattern would keep the bug. Several of them call _shared_insert() more than
once per request (qwen3 four times, gemma4e three, qwen3vl three, qwen3_8mtp
three); whether any of those calls throws its draw away needs checking one by
one, and this PR does not claim to have done it.

Test Plan

Qwen3.5-2B, two prompts that differ in one clause, temperature: 0, the same
build settings before and after, both sampling paths:

• top_k=40, temperature=0 -- the sampled path
• top_k=1, temperature=0 -- the greedy path, through sample_greedy()

Test Result

Before, both paths and both prompts answered B. After, all four answer A:

        before   after
Q k=1     B        A
Q k=40    B        A
R k=1     B        A
R k=40    B        A

A is the argmax of the unpenalised distribution (0.339 against 0.299 for
B), so the answer is no longer the penalised one.

For text generation the change is visible but harmless: see the two sentences
in the motivation. Nothing else moves -- record_in_history defaults to
true, and only the two Qwen3.5-VL paths pass false.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.

🤖 Generated with Claude Code

[manueldomke]

Rebased onto main after #790: record_in_history now sits next to the
sampler's new allowed mask (_sample() -> sampler->sample()). No change in
behaviour; answers on Qwen3.5-2B (NPU2) are identical to the pre-rebase build,
and repeated identical requests pass.