Repository navigation
fix(localmodel): ask the attested server for one token - #645
Open
ion-alpha-dev wants to merge 1 commit into
Open
ion-alpha-dev wants to merge 1 commit into
ion-alpha-dev wants to merge 1 commit into
Conversation
The listener handshake ends with a chat request carrying the probe key. The server accepts it, so it reaches the model, and the body set no max_tokens. A chat model given an empty conversation then writes a full answer of its own: gemma4:12b on an RTX 4070 Ti took 42 seconds, far past the 5 second bound, so every launch on that box was reported as "the listener is not the server that was launched" and `flynn models probe` and `flynn init --local` could not finish. Send max_tokens 1. The same request then returns in about 0.1 seconds. The first two requests are refused before the model, so they were never affected. The new test fakes a server that generates until max_tokens and fails with the reported error when the cap is removed. Signed-off-by: Ion Alpha <[email protected]>
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The last request of the listener handshake (the one carrying the probe key) now sends
"max_tokens":1, so the server answers without running a full generation.Why
That request is accepted, so it reaches the model, and it set no token limit. A chat model given an empty conversation writes a full answer of its own:
gemma4:12bon an RTX 4070 Ti (Windows, llama.cpp b9813 CUDA 13.3) took 42 seconds, past the handshake's 5 second bound. Every launch on that machine failed with "the listener is not the server that was launched: could not check ... Client.Timeout exceeded", soflynn models probeandflynn init --localcould not finish even though the server loaded the model in 7 seconds.How to verify
go test ./localmodel/ -run TestAttest.TestAttestAsksTheModelForOneTokenfakes a server that generates untilmax_tokens; with the cap removed it fails with the same timeout error users saw.--api-key: the probe-key request took 42.5 s without the cap and 0.11 s with it. A build of this branch gets past the handshake forflynn models probe gemma4:12bon the same box.Notes for reviewers
The first two handshake requests are refused at the auth check before the model is touched, so they were never affected. With this fix,
models probegets further and hits a separate problem in the context-window measurement (a prompt larger than the served context aborts the probe). That has its own change.