[manueldomke]

Motivation

/v1/chat/completions and /v1/completions write a hard-coded
{"logprobs", nullptr} into their responses. The sampler already keeps the
top-k of every sampled token -- token id, logit and probability -- but nothing
carries them to the response.

A model that answers with a single token -- a classifier, a router, a decision
model -- is unreadable without that distribution. The caller gets the chosen
option and cannot tell whether the model was certain or guessing. Three
Qwen3.5 fine-tunes converted for the NPU hit exactly this, and each of them
was built and measured against this patch:

• seimanu/decider-0.8b-NPU2
• seimanu/decider-2b-NPU2
• seimanu/decider-4b-NPU2

Their model cards carry the flm commit they were tested with and the
configuration that produced the numbers below.

Technical Details

Builds on #776. Until that one is merged this PR shows both commits; it exists
because without the fix the reported distribution would include the penalty of
a sample the model never used.

sampler.hpp/.cpp    capture_if_requested(), log_sum_exp_all() and a table of
                    exp() for every bf16 value; the sampling paths capture
                    once per draw while a request has the sampler armed
automodel.hpp/.cpp  top_logprob_t, logprobs_entry_t,
                    _request_logprobs_capture() and _record_top_logprobs()
modeling_qwen3_5vl  records where a token reaches the caller's text, not where
                    it was drawn
rest_handler.cpp    reads the request, builds both response shapes

One entry per token of message.content, in the same order, so a client
that pairs entries with tokens stays aligned. Recording happens where the token
is written to the response, which is what keeps the two in step: a token the
engine filters out gets no entry, and neither does EOS. A 17-token answer comes
back with 16 entries. An answer that carries a tool call gets null, because
message.content is empty and there are no tokens to describe.

logprob is a real log-probability -- logit minus logsumexp over the whole
vocabulary, after the penalties and before the temperature. It is not
renormalised over the entries listed, so those do not sum to 1; the rest of the
vocabulary lies outside the list.

The logsumexp is exact and cheap. The logits arrive as bf16 from the
engine, so there are only 65536 possible values, and a table of exp() for
each of them (512 KB, fits in L2) replaces a quarter of a million exp() calls
with as many reads. Values the penalties moved off the bf16 grid take the exact
call. The result differs from the shifted computation by about one float ulp,
which is rounding, not approximation. For the alternatives, N <= top_k needs
no sort at all: top_k_logits is already sorted by logit with the same
tie-break, so its beginning is the top-n.

The two request shapes differ, as they do upstream: chat takes logprobs
as a boolean plus top_logprobs (0-20), completions takes logprobs as an
integer 0-5 and has no top_logprobs. For completions 0 is a valid request: it
asks for the drawn token's own logprob with no alternatives.

A request that does not ask pays nothing. The sampler is disarmed at the
start of every request, before anything that can return early; it is armed only
in Qwen3.5-VL's generate(), and only when the request asks for logprobs.

Seen while testing this, not touched here: a model that emits half a
multi-byte character crashes the endpoint. nlohmann::json throws while
serialising the response text --

[json.exception.type_error.316] incomplete UTF-8 string; last byte: 0x98

-- and the request fails with a 500. It happens without logprobs too, so it
is not from this patch; the guard in _record_top_logprobs covers the
logprobs path, and the response text has no equivalent.

Limitations, unchanged by this patch:

• only Qwen3.5-VL reports; every other model still returns null
• streaming responses carry no logprobs
• with think: true the answer gets null: the tokens of the reasoning block
  land in reasoning_content, not in content
• think is a setting on the model, not on the request. One request with
  think: true leaves every later one in that mode, and they all come back with
  logprobs: null until it is set back. That is existing behaviour, but this
  patch is what makes it visible

With a decoding constraint (response_format, tool_choice, since #790)
the capture sees the masked logits, the ones the draw used, so logprob is
normalised over the tokens the constraint allows. Alternatives the constraint
forbids sit at -inf and are not reported, since -inf has no JSON number.
Tested with a json_schema response format on the sampled and the greedy
path: valid JSON, every reported logprob finite.

Test Plan

• Shape and alignment: a multi-token answer, entry count against token
  count; a tool-call answer; think: true
• Invariant: top_k=2, temperature=1, 50 runs -- the drawn token must
  appear in its own entry every time
• Non-ASCII: a German prompt producing umlauts, then rebuilding each token
  from its bytes and comparing
• Regression: the same request with and without logprobs, text compared
  character by character; then a run of without -> with -> without, to show the
  sampler is disarmed again
• Cost: wall clock, 300 tokens, nothing else running

Test Result

17-token answer          16 entries          (EOS gets none)
tool call                logprobs: null
think: true              logprobs: null
top_k=2, temp 1, 50x     50 of 50 in the list
umlauts                  bytes 0-255, every token rebuilds from its bytes
without logprobs         field is null, text unchanged
without -> with -> without   10.23 / 9.98 / 10.25 s, 300 tokens each

The cost is inside the run-to-run spread -- the three runs without logprobs
differ by 0.05 s on their own, and the run with logprobs was faster than the
first. The honest figure is: at most about 1 %, and only for requests that ask.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.

🤖 Generated with Claude Code

[manueldomke]

Rebased onto main after #790. The sampler's new allowed mask and
record_in_history are passed side by side; no change in behaviour beyond
the constraint handling described above. Re-tested on Qwen3.5-2B (NPU2):
answers, logprobs, repeated identical requests and a json_schema request
all pass.