Repository navigation
fix(models): end the probe context ladder at the served context - #646
Merged
Merged
Conversation
`flynn models probe` aborted after about two minutes when its context ladder sent a prompt larger than the context the local server was started with: llama.cpp answered HTTP 400 "exceeds the available context size" and MeasureEffectiveContext returned that as fatal, so no profile was written. Three changes: The OpenAI-compatible client now recognises a refusal for length (the llama.cpp exceed_context_size_error type, vLLM's "maximum context length", OpenAI's context_length_exceeded code) and wraps the new llm.ErrContextOverflow around it, so callers use errors.Is rather than matching message text. The error still prints as before. MeasureEffectiveContext treats that error as the end of the ladder, the same as a failed recall, and keeps the last length that passed. Any other error still fails. It also sizes later rungs from the token count the server reports for the first prompt (the filler runs near 3.5 characters per token, not the 4 assumed) and never stores more than the server counted, so a pass is no longer overstated by about 14%. The probe reads the served context from the server's /props and ends the ladder there, less room for the answer. Signed-off-by: Ion Alpha <[email protected]>
The declaration-comment length test fails on a doc comment of 15 lines or more. MeasureEffectiveContext was at 17; trim it to the invariants a reader of go doc needs. Signed-off-by: Ion Alpha <[email protected]>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
flynn models probenow finishes when its context ladder reaches a prompt the server refuses as too long, and no longer stores a context larger than what it sent. The OpenAI-compatible client classifies a context-overflow refusal into a new exported error,llm.ErrContextOverflow, which the ladder treats as its end.Why
On a machine where llama.cpp fits the context to the GPU, the server was started with 115456 tokens while the catalog advertises more. The ladder climbed past it, the server answered HTTP 400 "exceeds the available context size", and the probe returned that as fatal after about two minutes, so no profile was written and
flynn init --localset no default. The prompt was also sized at 4 characters per token regardless of the tokenizer, so a pass at a rung was recorded as a length the prompt did not have.How to verify
./dev/test ./llm/... ./internal/reliability/... ./cmd/flynn/...and./dev/lint. The new tests areTestContextOverflowIsTypedandTestContextOverflowKeepsMessage(client classification for llama.cpp, vLLM and OpenAI wording, and that a 5xx or 429 is not reclassified),TestMeasureEffectiveContext*andTestContextCandidatesStopAtServedContext(ladder), andTestProbeSurvivesServedContextBelowAdvertisedandTestServedContext(probe). Each fails when its cause is removed.Live on Windows 11, RTX 4070 Ti 12 GB, llama.cpp b9813, gemma4:12b Q4_0 (with the handshake change from #645 applied locally):
flynn models probe gemma4:12bcompleted in 166 seconds with tool calls, schema and instructions at 100% and a stored effective context of 65525 of 262144 advertised. The last rung (115200, inside the served 115456) was sent and the model did not recall the pass phrase, so the stored value is the 65536 rung as the server counted it.Notes for reviewers
llm.ErrContextOverflowonly../dev/apidiffon my machine also lists 51 incompatible changes unrelated to this diff (json.RawMessagereported asjsontext.Value), none touching this change./propsby a new helper incmd/flynn/modelprobe_ctx.go. If it is unavailable the ladder ends at the server's refusal instead, so vLLM and other servers rely on the typed error alone.probeAndStorekeeps its signature and delegates to a newprobeAndStoreWithin, to leave other tests that call it untouched.