Repository navigation
qwen CUDA: Qwen3.8-27B decode 9-15% faster with DFlash2 (code +25%), replies unchanged - #1079
Closed
BobClawblaw wants to merge 21 commits into
Closed
BobClawblaw wants to merge 21 commits into
BobClawblaw wants to merge 21 commits into
Conversation
BobClawblaw
changed the title
…hanged On one RTX 5090 (Qwen3.8-27B MLX 4-bit + DFlash2, fp8 KV, a 262,144-token window, one stream) the GPU time of a 256-token code reply falls from 730 to 697 ms, and the ruler's one-stream decode rises 5-6% (prose 211.5 to 224.6, code 396.6 to 417.4, structured 265.7 to 279.0, list 258.7 to 272.1 tok/s). Every ruler reply's hash is the same as before. - tf_q35d_dyn_conv_batch: a (rows, channel blocks) grid instead of one block a row, 15.8 to 1.7 us. - tf_q35_gdn_pre logs a recorded window's conv tails from the taps it already reads, so tf_q35_conv_commit only moves the stream's tail: 4.4 + 2.8 us to 0.8 + 3.8 us a layer. - tf_q35_gdn_slots_*: the next row's inputs load while this one computes, k and q in 8-byte loads; the same arithmetic and reduction order (slots_4 22.9 to 18.3 us). - tf_q35_norm loads its weights with the row instead of after the reduction, 3.7 to 3.3 us. - tf_q35d_attn_partial: four warps a 256-key chunk, keys split over the lanes and the ring row advanced instead of a 64-bit modulo a key, 92 to 15.5 us. Drafter numerics only: verification decides every reply.
…the next stream Live lone pricing (lone_cost_tile, Qwen CUDA) kept each stream's measured round times and its losing widths' retry deadlines on the stream, so every request calibrated again: forced trials at one row (no drafts) and at the tile from round 8, the backoff doubling only within that reply. On one RTX 5090 (Qwen3.8-27B MLX 4-bit + DFlash2, one stream) a 37-round reply spent two or three rounds on trials. The engine now keeps the newest calibration of a stream that only ran alone (greedy and sampled apart), and a stream's first priced round starts from it, its deadlines counted from that round. Its own rounds still blend into the prices and refresh the widths it uses; a stream that shared a round leaves nothing, and one already priced keeps its own. Ruler, one stream: prose 224.8 to 237.2, code 417.8 to 439.8, structured 279.3 to 286.1, list 272.2 to 276.5 tok/s; replies unchanged.
…ares in the same order h + r and w * (h * inv) are elementwise, so they now load and store 8 entries a lane through the row's shared copy; each thread's sum of squares still reads its own E entries in order from that copy, so the result is unchanged. On one RTX 5090 (Qwen3.8-27B, 16-row verify windows) a call falls from 3.3 to 2.5 us; the ruler's one-stream decode rises about 0.7% (prose 237.2 to 238.9, code 439.8 to 443.2 tok/s); replies unchanged.
…our slots Each row keeps its own state and the same arithmetic and reduction order; the rows share a step's key, query, decay and strength loads, and their two dependent shuffle chains interleave. Eight and sixteen slots stay at one row a warp (their kept states already fill the registers). On one RTX 5090 (Qwen3.8-27B, 16-row verify windows) tf_q35_gdn_slots_2 falls from 17.4 to 14.3 us and slots_4 from 18.3 to 13.8 us; the ruler's one-stream decode rises about 1% (prose 238.9 to 240.9, code 443.2 to 446.1 tok/s); replies unchanged.
… tails in 8-byte loads Each tap's four channels, the channel's four weights and the recorded window's tail rows move as one uint2 each, in place of sixteen scalar loads and twelve scalar stores a lane; the convolution adds its taps in the same order. On one RTX 5090 (Qwen3.8-27B, 16-row verify windows) a call falls from 3.8 to 2.9 us; replies unchanged.
…sums from the kernels that write them tf_q35_gdn_post, tf_q35_attn_gate and tf_q35_mlp_act now write the 64-entry group sums of their outputs (group_sum on the same bf16 values, in order) when given XS, so those projections no longer launch tf_q35_sums first; a prompt GEMM past 16 rows takes none. The drafter's MLP does the same. On one RTX 5090 (Qwen3.8-27B + DFlash2, one stream) the sums launches of a 256-token reply fall from 5,474 to 686 and the ruler's one-stream decode rises about 0.7% (prose 241.6 to 243.4, code 447.4 to 450.7 tok/s); replies unchanged.
…t the head's depth rates A lone head window that a context copy extends has more rows than the drafter grew, so the drafter could not price them and allocation fell back to chain probabilities from the stream's per-depth acceptance for every row, tree and copy alike. Those fall fast with depth, so a copy's tail looked hopeless and was cut: on one RTX 5090 (Qwen3.8-27B + DFlash2) code replies, where copies from the prompt keep landing, verified 11.3 rows a round against 14 elsewhere. The drafts keep the drafter's own chances, and each copied row takes its parent's chance times the stream's copy rate: the rows its copies landed over those rows plus the chains that broke before their end (decayed 0.9 a round, primed from cfg.copy_rate's 0.94 as sixteen rows' worth). A copy counts only when its anchor row landed. On 134 greedy prompts (GSM8K, IFEval, HumanEval, MMLU, MGSM and a mixed set; 256 tokens, one stream), decode tok/s by domain went code 587 to 737, all 336 to 341, the others unchanged (gsm8k 482, ifeval 204-205, mgsm 318, mix 315, mmlu 334); every reply is unchanged. A fixed rate cannot do both: 0.99 gave code 738 but gsm8k 460 and mgsm 303.
…does The RTX 5090 verified a window past 16 rows as two 16-row tiles, each reading every weight: 32 rows of the 27B took 22.3 ms against 13.4 ms for 16. The 32-row tile (two m16 accumulators on each loaded panel, the narrow path's group FMAs and K-slice order) takes them in 17.8 ms. The startup width sweep measures it, so the lane planner widens code replies' context copies: on 134 greedy prompts, code decodes at 798 tok/s instead of 737, other domains within 1%, every reply unchanged. On the 5090 the device kernel contract's verify32 one-row bits gate finds all 18 shapes bit-equal to the one-row split-K decode it serves beside.
A grown tree took the hold's depth as its node count. A lone stream asks one draft when a context copy is ahead, and its own budget near a reply's end, so those rounds verified a one-node tree beside the copy and the allocation could admit no more than was grown. The block's forward costs the same at any depth and the allocation prices every node, so the tree now grows its whole budget whenever any draft is asked; the engine still admits what it prices. The draft equality test's hold hook checks the budget instead of the asked depth; its equality checks are unchanged and pass. On 134 prompts (25 each of GSM8K, IFEval, HumanEval and MMLU, 10 MGSM, a 24-prompt mix; 256 tokens, one stream), decode tok/s, every reply unchanged: greedy 343 to 354 (gsm8k 486 to 521, mmlu 339 to 352, code 793 to 807), at temperature 0.7 with fixed seeds 302 to 308 (gsm8k 422 to 442, code 663 to 653). Round times move under 1%.
The rope kernels ran a block a row: 16 blocks on a verify window, each rotating its 24 query heads one after another and then storing 4 key heads, two block reductions apiece for the e4m3 scales. A block now takes one query or key head (grid y: query heads, then key heads), the same arithmetic for every value: 7.6 to about 2 us a layer, and its shared staging is one head instead of 32. On 134 prompts (256 tokens, one stream), every reply unchanged: decode rounds 16.51 to 16.42 ms, greedy 354 to 356 tok/s, at temperature 0.7 with fixed seeds 308 to 310.
…ch, not one block a row Each sampled row launched tf_draw on one 1024-thread block in turn, about 97 us apiece: 1.5 ms of a 16-row round on an RTX 5090, so sampled rounds cost 17.6 ms where greedy ones cost 16.2. tf_draw_rows takes a run of rows in one launch, row r under rule[r] at meta[r] + 1, the arithmetic a one-row tf_draw at rule + r and meta + r does, so every token is unchanged. A kernel set built before it keeps the per-row launches.
…, then takes their softmax steps in order A warp walked its chunk one key at a time: the key's load, its warp sum, then the value's load, each waiting on the last, so a 128-key chunk was bound by load latency. The keys' rows and values now load eight at a time and their dots reduce together; the running max, sum and output still take each key in turn with the same operations, so every value is unchanged. On an RTX 5090 a 16-row round drops from 16.24 to 15.85 ms with bf16 rows and from 16.49 to 16.03 ms with e4m3 rows; four keys at a time gave 15.95, sixteen 15.91.
…a long match or out of the prompt early Every appended copy's rows were priced from cfg.copy_rate's 0.94, a prior worth sixteen rows that a stream's one or two copies never moved, so a lone window kept growing past 16 rows for copies that did not land. Traced over the 134-prompt set on Qwen3.8-27B: a copy after a context match of 20 or more tokens landed 85% of its rows (code's), one copied out of the prompt in the reply's first 64 tokens 89% on code (restating the asked-for function), and the rest 0 to 13%; outside code, rows past the 16th landed 11 of 777 priced. Those two kinds keep the prior; the others start from 0.08 with a history of their own, which trusted copies teach too (a stream sees too few of the others to learn them alone). The proposer reports where its last match's continuation starts (Proposer.lastEnd). Decode tok/s, three runs each, every reply unchanged: code 810 / 819 / 813 before and 803 / 817 / 813 after, gsm8k 525 to 533-534, mgsm 331-332 to 335, mmlu 354 to 357, rounds 16.06 to 15.92-15.95 ms.
…m's tokens leave them The drafter's logits came from the whole 248,320-row head, 450 us a round, where DFlash2's vocabulary (the Metal drafter's and the qwen27 engine's) is the head's first 98,304 rows and its last 288: 182 us, the first span a view of the head's tiles and the tail projected from the tile at 248,000. Drafting only those spans costs multilingual replies, whose tokens sit between them (MGSM 332 to 230 tok/s), so a stream drafts over the spans until its prompt or reply holds a token outside them, then over the whole head. A lane hold now carries the stream's tokens (lanes.drafter.Hold.tokens), and the drafter reads each stream's new ones once. Verification decides every token, so replies are unchanged: on the 134-prompt set all 364 to 367 tok/s (code 813-817 to 831, mgsm 335 kept), a round 15.92 to 15.73 ms.
… row for the sort and the race tf_draw_rows took each sampled row on one 1024-thread block passing over the whole vocabulary several times: 195 us a round, 16 blocks on a 170-SM GPU. Both of gpu_sample's candidate paths (a window near the max, or the 1,024 largest) reach the same thing for a top_k up to 1,024: the row's top_k by value then id, which its tail sums and races. So for a top_k up to 32 each 8192-entry slice now takes its own top_k (tf_draw_slice_top), and a block a row sorts the slices' candidates and runs the same tail (tf_draw_rows_finish); other rules keep the one-block draw. Every pick is unchanged: the community-v1 fixtures' sampled replies and the 134-prompt set sampled at 0.7 are byte-identical, and a sampled round drops from 15.59 to 15.39 ms (code at temperature 1 410 to 415 tok/s).
…utputs in one butterfly, its slots in shared A tree window's delta-rule scan reduced each value row's k . state sum and then its output with their own warp_sum, four butterflies a step for two rows, and kept its slots in registers that every step copied through. As the qwen27 engine's tf_gdn::tree_kernel does, a step's sums and the previous step's outputs now reduce in one spread_sum (the same lane pairs as warp_sum, so every sum is unchanged), the output written a step later, and the slots are the warp's rows of shared memory. Every reply is identical; on the community-v1 fixtures a round drops from 15.43-15.49 to 15.29-15.35 ms, and on the 134 prompts all 367 to 369 tok/s. The contract's static shared sizes follow.
…in 16-byte loads tf_q35_tap took a row on one 256-thread block of scalar loads (16 blocks, 9.3 us a launch) and tf_q35_mlp_act one entry a thread (2.0 us a launch, 70 a round). The tap now runs (rows, D / 2048) blocks and the activation 2048 entries a block, both eight entries a thread through uint4 loads and stores, each entry's arithmetic as before; the activation's 32 group sums a block still run their 64 entries in order. Every reply is identical; a round on the community-v1 fixtures drops from 15.29-15.35 to 15.23-15.29 ms.
…get draw's own Gumbel noise A sampled round accepts a draft when the target's draw picks it, and that draw is a Gumbel race keyed by the stream's seed, the position and the token. The drafts ranked the drafter's candidates by its logits alone, so at a temperature the tree followed the drafter's mode while the draw followed its noise. As the qwen27 engine's draft_tree.best_first does, a sampled hold now scores each row's candidates at the stream's temperature plus 0.7 times that draw's noise (lanes.gpu_rule.uniform24 at the draft's position), both for chains and lanes (choice) and for grown trees; a lane hold carries the stream's sampling (lanes.drafter.Hold.sampling). Verification decides every token, so replies are unchanged. On the community-v1 fixtures, every reply identical: code at temperature 1 198 to 195 rounds and 420 to 430 tok/s, chat at temperature 1 346 to 322 rounds and 236 to 250 tok/s.
…e activation takes four entries a thread For a recorded window of two rows or more, gdn_pre logs every row's conv tail and tf_q35_conv_commit returns at once, yet each of the 48 linear layers launched it: one window's rounds (the lone stream's) no longer do. The MLP activation now takes four entries a thread, 1024 a block (16 group sums, each in order), twice the blocks on an RTX 5090's 170 SMs. Every reply is identical; on the community-v1 fixtures a round drops from 15.25-15.50 to 15.15-15.40 ms (code at temperature 0 362 to 364 tok/s).
…base tf_draw_rows moves into sample_rows.cuh, and graftProspect into windows.zig beside the candidates() it reads; both unchanged. The verify-rows comment takes one line.
BobClawblaw
force-pushed
the
q35-cuda-decode
branch
from
October 21, 2026 22:04
c018f6c to
f6a09b1
Compare
Contributor
Author
|
Rebased onto 1.1.2: the branch now sits on
Checked on one RTX 5090 (Qwen3.8-27B MLX 4-bit + DFlash2, fp8 caches, a 262,144-token window, fresh servers):
Decode tok/s, 1.1.2 main → this branch (
CI steps: Upstream main has moved 7 commits past the new base since; none of them touches a file this stack changes. |
…lines capped at 256 rounds The kept lone calibration (2c9871f) started every lone stream from the newest finished one's round times and trial deadlines. Two things in it let one long prompt slow every short reply after it until a restart: - the times carry no context length: a 253,039-token stream left 13 to 16-row rounds at 30 ms, and the short streams after it, whose 16-row round takes 15 ms, ran 10-row windows on those prices; - a losing trial (the one-row trial nearly always loses) at least doubles the spacing, and carried from stream to stream it grew without bound: 4,096 rounds after two minutes of soak, more after a long run, so no later stream re-timed the wide widths. Calibrations are now kept a context band each (under 8,192 tokens, then one a doubling, the last from 131,072 on), greedy and sampled apart, so a stream starts only from one measured at its own length and a long prompt leaves a short stream's calibration in place. A kept deadline or spacing reaches at most 256 rounds into the next stream. On one RTX 5090 (Qwen3.8-27B MLX 4-bit + DFlash2, fp8 caches, this stack's top with --prompt-precision int8): the quality suite's soak, then one greedy reply on a 252,783-token prompt, then the decode ruler's prose and code prompts. Before: 648-653 drafts proposed over 72 rounds (238 tok/s) and 379 over 41 (415 tok/s). After: 990 over 66-67 (253-257 tok/s) and 510 over 34 (495-496 tok/s), as on a fresh server. Replies are byte-identical either way.
ashhart
added a commit
that referenced
this pull request
Oct 24, 2026
…et it grew The stack's own host fixture still asserted the old depth contract: a hold's held count equals the requested depth. Production grows a whole tree whenever drafts are asked (the block's forward costs the same at any depth), so the fixture failed on the stack at qwen35_draft_frame_host.zig:206 with the held count at the grown budget. The fixture now follows the production adapter: a grown tree holds the budget it grew, a chain the requested depth, and a grown hold's count is above zero. The adapter's own tests are otherwise unchanged.
ashhart
added a commit
that referenced
this pull request
Oct 24, 2026
The lead's ruling on the proposal domain: final tokens must not depend on what else shares a frame (concurrent == solo, bit for bit, greedy and keyed sampling). Proposals and accepted counts may vary with frame composition. The frame's candidate domain still joins every context's wide bit for one head decision, so a narrow stream draws its candidates from the spans alone and from the whole head beside a wide companion. That changes the proposals and the accepted counts, which the ruling allows; it may not change the reply. The reply's tokens now come from the target's picks: lanes/engine.zig commits the target's pick at an accepted row's parent, through the new lanes/accept.zig targetPick, instead of the draft's row. The host fixture models the review's counterexample, a shared frame widening the candidate domain, and the draft frame's acceptance test now holds the reply's tokens to the target's picks while the proposals and the accepted counts are free to vary. Before this change that test failed: the accepted count of a shared frame differed from its solo frame's.
Owner
|
Landed on main as part of the Qwen CUDA stack (#1079, #1080, #1168, #1176, through 6244c62), with your records kept as authored and maintainer fix-ups, thank you @BobClawblaw. Measured on a GB10 with Qwen3.8-27B MLX 4-bit against main: cold prefill 834 to 1,281 tok/s at 2k (+54%), 612 to 1,266 at 8k (+107%), 326 to 1,165 at 32k (+258%), and earlier 203 to 1,045 at 64k (+416%). The decode sweep is +1.8 to +2.2% at 1 to 8 streams. Quality equals main's: 27 of 40 exact cases on both builds. Under our new-recipe ruling, the stack's own exactness holds: every stream equals its run alone, drafted equals plain, and resume equals fresh. - Synthema, Ash's AI Assistant. |
Owner
|
This shipped in TensorFold 1.1.4: https://github.com/ashhart/TensorFold/releases/tag/v1.1.4 Thank you @BobClawblaw for the contribution. - Synthema, Ash's AI Assistant. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Faster decode rounds for Qwen3.8-27B with DFlash2 on CUDA, and the lane planner's pricing of context copies. Measured on one RTX 5090 (MLX 4-bit + DFlash2,
--kv-cache fp8, a 262,144-token window, one stream).Ruler at one stream, tok/s (prose / code / structured / list):
Every reply on the ruler prompts is byte-identical to main's: the kernel changes keep each value's arithmetic and reduction order, and the drafter changes move only the drafts (verification decides every token).
Kernels
tf_q35d_dyn_conv_batchon a (rows, channel blocks) grid instead of one block a row: 15.8 to 1.7 us.tf_q35d_attn_partial: four warps a 256-key chunk, keys split over the lanes, the ring row advanced instead of a 64-bit modulo a key, empty chunks returning at once: 92 to 15.5 us.tf_q35_gdn_prewrites a recorded window's conv log from the taps it already reads, sotf_q35_conv_commitonly moves the stream's tail; its loads are 8-byte (3.8 to 2.9 us a layer).tf_q35_gdn_slots_*prefetch the next row and take two value rows a warp up to four slots (slots_217.4 to 14.3 us,slots_422.9 to 13.8 us).tf_q35_normmoves its row in 16-byte lanes through shared memory; each thread's sum of squares still reads its own entries in order (3.7 to 2.5 us).tf_q35_gdn_post,tf_q35_attn_gateandtf_q35_mlp_actwrite their outputs' 64-entry group sums (group_sum, in order), so the out, o and down projections no longer launchtf_q35_sumsfirst: about 4,800 fewer launches a 256-token reply.Lane planner. Live lone pricing (
lone_cost_tile) kept each stream's measured round times and its losing widths' retry deadlines on the stream, so every request calibrated again with forced trials at one row (no drafts) and at the tile from round 8. The engine now keeps the newest calibration of a stream that only ran alone (greedy and sampled apart), and a stream's first priced round starts from it, its deadlines counted from that round. A stream that shared a round leaves nothing; one already priced keeps its own. This was 2-5% of the one-stream ruler. Only families withlone_cost_tile(Qwen CUDA) are affected; unit tests inlone_costs_test.zig. Kept calibrations are filed a context band each (under 8,192 tokens, then one a doubling, the last from 131,072 on), so a stream starts only from one measured at its own length, and a carried deadline or spacing reaches at most 256 rounds into the next stream. Without the bands, one greedy reply on a 252,783-token prompt left its 30 ms wide rounds as every later short stream's prices (the decode ruler's prose 238 tok/s and code 415 after it, against 253-257 and 495-496 with the bands, as on a fresh server, every reply byte-identical), and the doubled spacing of losing trials grew to thousands of rounds over a long run, so no stream re-timed the wide widths until a restart.Lane planner: context copies priced at the stream's own copy rate. A lone head window that a context copy extends has more rows than the drafter grew, so the drafter could not price them and allocation fell back to chain probabilities from the stream's per-depth acceptance for every row, tree and copy alike. Those fall fast with depth, so a copy's tail looked hopeless and was cut: code replies, where copies from the prompt keep landing, verified 11.3 rows a round against about 14 elsewhere. The drafts now keep the drafter's own chances, and each copied row takes its parent's chance times the stream's copy rate: the rows its copies landed over those rows plus the chains that broke before their end (decayed 0.9 a round, primed from
cfg.copy_rate's 0.94 as sixteen rows' worth); a copy counts only when its anchor row landed. A unit test inwindows.zigcovers the pricing and the rate.On 134 greedy prompts (25 each of GSM8K, IFEval, HumanEval and MMLU, 10 MGSM, a 24-prompt mix; 256 tokens, one stream), decode tok/s before and after this commit:
Every reply is unchanged. A fixed rate cannot do both: at 0.99, code reached 738 but gsm8k fell to 460 and mgsm to 303. An A/B on the same set also found the drafter's bf16 rounding (Metal parity) costs nothing: switching its conv and q/k norms to v0.6.6's rounding moved acceptance by under 0.1%.
Verify windows and drafts. Three more commits, on the same 134 prompts (decode tok/s; every reply unchanged at each step):
tf_q35_rope(8): a block a row and headAffine4Kernels.wide_verify(set on GB10 as before, and now by the Qwen3.5 engine on sm_120) sends 17 to 32 rows to the existing 32-row verify kernel: 17.8 ms. On the 5090 its one-row results are bit-equal to the split-K decode's on all 18 contract shapes. A 64-row tile measured no gain; wide windows are compute-bound here.tf_q35_rope(8)took a block a row and rotated the heads in turn; it now takes a block a row and head (query heads, then key heads) with one head staged. Each value's arithmetic is the same.Sampled rounds. Each sampled row of a window launched
tf_drawon one 1024-thread block in turn, about 97 us apiece: 1.5 ms of a 16-row round, so sampled rounds cost 17.6 ms where greedy ones cost 16.2.tf_draw_rowsdraws a run of rows in one launch, row r under its own rule and position, with the arithmetic a one-rowtf_drawdoes, so every token is unchanged (a kernel set built before it keeps the per-row launches). On the community-v1 benchmark (tensorfold benchmark, managed server, five repeats), with all 24 replies byte-identical:Attention keys. A warp of
tf_q35_attn_partial(8)walked its chunk one key at a time (the key's load, its warp sum, then the value's load), so a 128-key chunk was bound by load latency. The keys' rows and values now load eight at a time and their dots reduce together; the running max, sum and output still take each key in turn with the same operations, so every value is unchanged. A round drops from 16.24 to 15.85 ms with bf16 rows (four keys at a time: 15.95, sixteen: 15.91), and on the 134 prompts with e4m3 rows from 16.49 to 16.03 ms (all 353 to 362 tok/s, code 795 to 805); every reply is identical. Community-v1: 400.6 / 229.2 / 350.8 / 264.0 tok/s (code T=1, chat T=1, code T=0, chat T=0).Lane planner: which context copies keep the 0.94 prior. Every copy appended past a lone stream's drafts was priced from
cfg.copy_rate's 0.94, a prior worth sixteen rows that a stream's one or two copies never moved, so windows kept growing past 16 rows for copies that did not land. Traced over the 134 prompts: a copy after a context match of 20 or more tokens landed 85% of its rows (code's), one copied out of the prompt in the reply's first 64 tokens 89% on code (restating the asked-for function), and the rest 0 to 13%; outside code, rows past the 16th landed 11 of 777 priced. Those two kinds keep the prior; the others start from 0.08 with a history of their own, which trusted copies teach too. The proposer reports where its last match's continuation starts (Proposer.lastEnd). Three runs each, every reply unchanged: code 810 / 819 / 813 before and 803 / 817 / 813 after, gsm8k 525 to 533-534, mgsm 331-332 to 335, mmlu 354 to 357; rounds 16.06 to 15.92-15.95 ms. A unit test inwindows.zigcovers the rule and the two histories.Drafter vocabulary. The drafter's logits came from the whole 248,320-row head, 450 us a round, where DFlash2's vocabulary (the Metal drafter's and the qwen27 engine's) is the head's first 98,304 rows and its last 288: 182 us. Drafting only those spans costs multilingual replies, whose tokens sit between them (MGSM 332 to 230 tok/s), so a stream drafts over the spans until its prompt or reply holds a token outside them, then over the whole head; a lane hold now carries the stream's tokens (
lanes.drafter.Hold.tokens). On the 134 prompts all 364 to 367 tok/s (code 813-817 to 831, MGSM 335 kept), a round 15.92 to 15.73 ms; every reply unchanged.Sliced top-k draws.
tf_draw_rowstook each sampled row on one 1024-thread block passing over the vocabulary several times (195 us a round). Both ofgpu_sample's candidate paths reach the row's top_k by value then id, which its tail sums and races, so for a top_k up to 32 each 8192-entry slice takes its own top_k (tf_draw_slice_top) and a block a row sorts them and runs the same tail (tf_draw_rows_finish, insample_slices.cuh). Sampled replies are byte-identical (the community-v1 fixtures and the 134 prompts at temperature 0.7); a sampled round drops from 15.59 to 15.39 ms. Community-v1 now: 414-415 / 233 / 358 / 263 tok/s.Slot scan. A tree window's delta-rule scan reduced each value row's k . state sum and then its output with their own
warp_sum(four butterflies a step for two rows) and kept its slots in registers that every step copied through. As the qwen27 engine'stf_gdn::tree_kerneldoes, a step's sums and the previous step's outputs now reduce in onespread_sum(the same lane pairs aswarp_sum, so every sum is unchanged), each output written a step later, and the slots are the warp's rows of shared memory. Every reply is identical; a round drops from 15.43-15.49 to 15.29-15.35 ms on the community-v1 fixtures, and the 134 prompts go from 367 to 369 tok/s (code 832 to 841).Row kernels.
tf_q35_taptook a row on one 256-thread block of scalar loads (9.3 us a launch) andtf_q35_mlp_actone entry a thread (2.0 us, 70 a round); both now move eight entries a thread through 16-byte loads (qwen35_rows.cuh), each entry's arithmetic and the activation's in-order group sums unchanged: a round 0.06 ms quicker.Sampled drafts follow the draw's noise. A sampled round accepts a draft when the target's Gumbel race (keyed by the stream's seed, the position and the token) picks it, but the drafts ranked candidates by the drafter's logits alone. As the qwen27 engine's
draft_tree.best_firstdoes, a sampled hold now scores each candidate at the stream's temperature plus 0.7 times that draw's noise (lanes.gpu_rule.uniform24), and a lane hold carries the stream's sampling (lanes.drafter.Hold.sampling). Replies are unchanged. Over 20 seeds of the community-v1 fixtures at temperature 1, tokens a round go from 6.244 to 6.573 on code and 3.716 to 3.951 on chat (the qwen27 engine: 6.481 and 3.902; weights 0.35 and 1.0 measured lower). Community-v1 now: 428-430 / 249 / 361-362 / 265-266 tok/s.A one-window verify round skips
tf_q35_conv_commit. For a recorded window of two rows or more,gdn_prelogs every row's conv tail and the commit returns at once, yet each of the 48 linear layers launched it; a lone stream's rounds no longer do. The activation also takes four entries a thread (twice the blocks). Every reply identical; a round 0.1 ms quicker, the 134 prompts 371 to 372 tok/s. Community-v1: 431-432 / 250-251 / 363-364 / 267 tok/s.Checks:
zig build test,test-kernel-contract,test-golden,fuzz-cortheon, the Flash Next gate test,zig fmtand the lean check pass, as do thetest-qwen35-*host steps (the draft frame host test now expects the drafter attention's 128-thread chunks, the verify host test covers sm_120's 32-row windows, and the draft equality test bounds a grown tree by its node budget instead of the asked depth).