Repository navigation
qwen CUDA: TF_DRAFT_NODES above 15 serves; a stop or cancel lands between prompt chunks - #1078
Closed
BobClawblaw wants to merge 2 commits into
Closed
BobClawblaw wants to merge 2 commits into
BobClawblaw wants to merge 2 commits into
Conversation
This was referenced Oct 21, 2026
… instead of failing every round With TF_DRAFT_NODES above 15 the server started, then every request failed with BadDraftFrame: the planner asks a grown tree for its node budget as the hold's depth, and the frame refused any depth past block - 1, the limit of a chain. Grown trees (block 16, no planner lanes) now take up to tree.nodes() (at most the 64-row window); chains and 8-row blocks keep block - 1. On one RTX 5090, TF_DRAFT_NODES=24 and 32 serve with every reply unchanged.
…t's halt The CUDA backend's prompt pass never asked Stream.isCancelled, so a stop signal during a long prompt waited for the whole pass: the host's halt joins the lane thread, which was still prefilling. With a 230k-token prompt in progress the server stopped draining and never exited; a client that left during a pass also kept the GPU until it finished. The pass now asks before each chunk, as the other backends do, and its cache goes back to the pool reset. On one RTX 5090 (Qwen3.8-27B, a 75,616-token prompt): SIGTERM 15 s into the pass exits after 23 s (the 20 s drain, then the next chunk), against 134 s before; the request ends "server restarting: retry shortly". A streaming client killed mid-pass ends its request within a second (client-left).
BobClawblaw
force-pushed
the
q35-cuda-fixes
branch
from
October 22, 2026 10:17
8e2f0bd to
4808c94
Compare
Contributor
Author
|
Rebased onto current main ( |
This was referenced Oct 24, 2026
Owner
|
Thanks @BobClawblaw. Both commits are on - Synthema, Ash's AI Assistant. |
ashhart
added a commit
that referenced
this pull request
Oct 24, 2026
Keep #1078's between-chunk cancellation point and save its completed prefix before returning Cancelled, with one check on an uncancelled path. Preserve main's recorder score forwarding when adding the optional cancellation hook. Add whole and stepped fake-lane controls proving unchanged uncancelled work. Carry Sapphire-Rapids' two code records with their authors, dates and messages. Retain only host-free opening README prose from the third record, attributed as historical contributor observations. Omit its raw receipt artifacts and artifact-dependent instructions; integrated GPU qualification remains owed. Fixes #1225. Refs #1228.
Owner
|
This shipped in TensorFold 1.1.4: https://github.com/ashhart/TensorFold/releases/tag/v1.1.4 Thank you @BobClawblaw for the contribution. - Synthema, Ash's AI Assistant. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two serving fixes for the Qwen3.5 CUDA family, found on one RTX 5090 (Qwen3.8-27B MLX 4-bit + DFlash2,
--kv-cache fp8, a 262,144-token window).TF_DRAFT_NODESabove 15 failed every request. The server started, then every round failed withBadDraftFrame. The planner asks a grown tree for its node budget as the hold's depth, andcuda_draft_frame.stagerefused any depth pastblock - 1, a chain's limit. Grown trees (block 16, no planner lanes) now stage up totree.nodes()(at most the 64-row window); chains and 8-row blocks keepblock - 1.TF_DRAFT_NODES=24and32serve with every reply unchanged. A host test inqwen35_draft_frame_host.zigcovers both limits and fails without the fix.A stop or a cancel waited for the whole prompt pass. The CUDA prompt pass never asked
Stream.isCancelled, so the host's halt (which joins the lane thread) waited until a long prefill finished: with a 230k-token prompt in progress the server stopped draining and did not exit, and a client that left mid-pass kept the GPU until the pass ended. The pass now asks before each chunk, as the other backends do; its cache goes back to the pool and is reset when taken again. With a 75,616-token prompt, SIGTERM 15 s into the pass exits after 23 s (the 20 s drain, then the next chunk) instead of 134 s, the request ending "server restarting: retry shortly"; a streaming client killed mid-pass ends its request within a second (client-left).The Metal
qwen3_5backend's prompt loop (backend.zig) has the same missing check; that is left to a Metal change.Checks:
zig build test,test-kernel-contract,test-golden,fuzz-cortheon, the Flash Next gate test,zig fmtand the lean check pass, as do thetest-qwen35-*host steps.