serving: a caller gets the characters the model wrote, not replacement characters - #1732
Merged
Merged
Conversation
…t characters A runtime decodes each token to text on its own, so a multibyte character split across two tokens arrives as U+FFFD in both deltas. Found in soak 7 on the blessed pin: a 384 token completion came back reading "between **<U+FFFD>m/2<U+FFFD>**" where the model had written the ceiling brackets, and the gateway handed that to the caller. The per-token bytes are unaffected, and the audit already binds the miner's ids to them. When every token reported its bytes and they decode cleanly, the completion is now assembled from the bytes; a partial or undecodable byte list leaves the deltas as sent. Streaming callers still receive the runtime's deltas verbatim - repairing those needs buffering across token boundaries, which is a separate change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found during soak 7, on the blessed pin, on a real 5090.
A runtime decodes each token to text independently, so a multibyte character split across two tokens reaches the stream as U+FFFD in both deltas. A 384-token completion came back as:
where the model wrote
between **⌈m/2⌉** and **m** children. The gateway returns that assembled text to the caller, so this is what a paying user receives.Reproduced directly against the pod: of 384 tokens, every one reported
bytes, none were missing, andb''.join(token_bytes)decodes to the correct text — only the concatenateddelta.contentis damaged. So the information was never lost, just discarded.StreamAssembler.text()now assembles the completion from the per-token bytes when every token reported them and they decode cleanly; a partial or undecodable byte list leaves the deltas exactly as sent. The audit already treats those bytes as ground truth (#1724 binds the miner's token ids to them), so this makes the text the caller sees agree with the bytes the validator verifies.This also removes one source of the
token ids do not spell the completionsoft miss: that check has to skip whenever either side carries U+FFFD, which is precisely when a character was split.Not fixed here: streaming callers still receive the runtime's deltas verbatim. Repairing those needs buffering across token boundaries so a split character is held until complete — a separate change with real latency implications.
Tests use the exact token shape captured from the live pod. CI green (ruff, format, pyright, vulture, 760 tests).