Repository navigation
qwen CUDA: long-prompt prefill 5.4x faster (4096-row chunks, tensor-core attention, the delta-rule chain) - #1080
Closed
BobClawblaw wants to merge 23 commits into
Closed
BobClawblaw wants to merge 23 commits into
BobClawblaw wants to merge 23 commits into
Conversation
BobClawblaw
force-pushed
the
q35-cuda-prefill
branch
9 times, most recently
from
October 20, 2026 21:02
e1e4427 to
2bd805a
Compare
This was referenced Oct 21, 2026
…hanged On one RTX 5090 (Qwen3.8-27B MLX 4-bit + DFlash2, fp8 KV, a 262,144-token window, one stream) the GPU time of a 256-token code reply falls from 730 to 697 ms, and the ruler's one-stream decode rises 5-6% (prose 211.5 to 224.6, code 396.6 to 417.4, structured 265.7 to 279.0, list 258.7 to 272.1 tok/s). Every ruler reply's hash is the same as before. - tf_q35d_dyn_conv_batch: a (rows, channel blocks) grid instead of one block a row, 15.8 to 1.7 us. - tf_q35_gdn_pre logs a recorded window's conv tails from the taps it already reads, so tf_q35_conv_commit only moves the stream's tail: 4.4 + 2.8 us to 0.8 + 3.8 us a layer. - tf_q35_gdn_slots_*: the next row's inputs load while this one computes, k and q in 8-byte loads; the same arithmetic and reduction order (slots_4 22.9 to 18.3 us). - tf_q35_norm loads its weights with the row instead of after the reduction, 3.7 to 3.3 us. - tf_q35d_attn_partial: four warps a 256-key chunk, keys split over the lanes and the ring row advanced instead of a 64-bit modulo a key, 92 to 15.5 us. Drafter numerics only: verification decides every reply.
…the next stream Live lone pricing (lone_cost_tile, Qwen CUDA) kept each stream's measured round times and its losing widths' retry deadlines on the stream, so every request calibrated again: forced trials at one row (no drafts) and at the tile from round 8, the backoff doubling only within that reply. On one RTX 5090 (Qwen3.8-27B MLX 4-bit + DFlash2, one stream) a 37-round reply spent two or three rounds on trials. The engine now keeps the newest calibration of a stream that only ran alone (greedy and sampled apart), and a stream's first priced round starts from it, its deadlines counted from that round. Its own rounds still blend into the prices and refresh the widths it uses; a stream that shared a round leaves nothing, and one already priced keeps its own. Ruler, one stream: prose 224.8 to 237.2, code 417.8 to 439.8, structured 279.3 to 286.1, list 272.2 to 276.5 tok/s; replies unchanged.
…ares in the same order h + r and w * (h * inv) are elementwise, so they now load and store 8 entries a lane through the row's shared copy; each thread's sum of squares still reads its own E entries in order from that copy, so the result is unchanged. On one RTX 5090 (Qwen3.8-27B, 16-row verify windows) a call falls from 3.3 to 2.5 us; the ruler's one-stream decode rises about 0.7% (prose 237.2 to 238.9, code 439.8 to 443.2 tok/s); replies unchanged.
…our slots Each row keeps its own state and the same arithmetic and reduction order; the rows share a step's key, query, decay and strength loads, and their two dependent shuffle chains interleave. Eight and sixteen slots stay at one row a warp (their kept states already fill the registers). On one RTX 5090 (Qwen3.8-27B, 16-row verify windows) tf_q35_gdn_slots_2 falls from 17.4 to 14.3 us and slots_4 from 18.3 to 13.8 us; the ruler's one-stream decode rises about 1% (prose 238.9 to 240.9, code 443.2 to 446.1 tok/s); replies unchanged.
… tails in 8-byte loads Each tap's four channels, the channel's four weights and the recorded window's tail rows move as one uint2 each, in place of sixteen scalar loads and twelve scalar stores a lane; the convolution adds its taps in the same order. On one RTX 5090 (Qwen3.8-27B, 16-row verify windows) a call falls from 3.8 to 2.9 us; replies unchanged.
…sums from the kernels that write them tf_q35_gdn_post, tf_q35_attn_gate and tf_q35_mlp_act now write the 64-entry group sums of their outputs (group_sum on the same bf16 values, in order) when given XS, so those projections no longer launch tf_q35_sums first; a prompt GEMM past 16 rows takes none. The drafter's MLP does the same. On one RTX 5090 (Qwen3.8-27B + DFlash2, one stream) the sums launches of a 256-token reply fall from 5,474 to 686 and the ruler's one-stream decode rises about 0.7% (prose 241.6 to 243.4, code 447.4 to 450.7 tok/s); replies unchanged.
…t the head's depth rates A lone head window that a context copy extends has more rows than the drafter grew, so the drafter could not price them and allocation fell back to chain probabilities from the stream's per-depth acceptance for every row, tree and copy alike. Those fall fast with depth, so a copy's tail looked hopeless and was cut: on one RTX 5090 (Qwen3.8-27B + DFlash2) code replies, where copies from the prompt keep landing, verified 11.3 rows a round against 14 elsewhere. The drafts keep the drafter's own chances, and each copied row takes its parent's chance times the stream's copy rate: the rows its copies landed over those rows plus the chains that broke before their end (decayed 0.9 a round, primed from cfg.copy_rate's 0.94 as sixteen rows' worth). A copy counts only when its anchor row landed. On 134 greedy prompts (GSM8K, IFEval, HumanEval, MMLU, MGSM and a mixed set; 256 tokens, one stream), decode tok/s by domain went code 587 to 737, all 336 to 341, the others unchanged (gsm8k 482, ifeval 204-205, mgsm 318, mix 315, mmlu 334); every reply is unchanged. A fixed rate cannot do both: 0.99 gave code 738 but gsm8k 460 and mgsm 303.
…does The RTX 5090 verified a window past 16 rows as two 16-row tiles, each reading every weight: 32 rows of the 27B took 22.3 ms against 13.4 ms for 16. The 32-row tile (two m16 accumulators on each loaded panel, the narrow path's group FMAs and K-slice order) takes them in 17.8 ms. The startup width sweep measures it, so the lane planner widens code replies' context copies: on 134 greedy prompts, code decodes at 798 tok/s instead of 737, other domains within 1%, every reply unchanged. On the 5090 the device kernel contract's verify32 one-row bits gate finds all 18 shapes bit-equal to the one-row split-K decode it serves beside.
A grown tree took the hold's depth as its node count. A lone stream asks one draft when a context copy is ahead, and its own budget near a reply's end, so those rounds verified a one-node tree beside the copy and the allocation could admit no more than was grown. The block's forward costs the same at any depth and the allocation prices every node, so the tree now grows its whole budget whenever any draft is asked; the engine still admits what it prices. The draft equality test's hold hook checks the budget instead of the asked depth; its equality checks are unchanged and pass. On 134 prompts (25 each of GSM8K, IFEval, HumanEval and MMLU, 10 MGSM, a 24-prompt mix; 256 tokens, one stream), decode tok/s, every reply unchanged: greedy 343 to 354 (gsm8k 486 to 521, mmlu 339 to 352, code 793 to 807), at temperature 0.7 with fixed seeds 302 to 308 (gsm8k 422 to 442, code 663 to 653). Round times move under 1%.
The rope kernels ran a block a row: 16 blocks on a verify window, each rotating its 24 query heads one after another and then storing 4 key heads, two block reductions apiece for the e4m3 scales. A block now takes one query or key head (grid y: query heads, then key heads), the same arithmetic for every value: 7.6 to about 2 us a layer, and its shared staging is one head instead of 32. On 134 prompts (256 tokens, one stream), every reply unchanged: decode rounds 16.51 to 16.42 ms, greedy 354 to 356 tok/s, at temperature 0.7 with fixed seeds 308 to 310.
…ch, not one block a row Each sampled row launched tf_draw on one 1024-thread block in turn, about 97 us apiece: 1.5 ms of a 16-row round on an RTX 5090, so sampled rounds cost 17.6 ms where greedy ones cost 16.2. tf_draw_rows takes a run of rows in one launch, row r under rule[r] at meta[r] + 1, the arithmetic a one-row tf_draw at rule + r and meta + r does, so every token is unchanged. A kernel set built before it keeps the per-row launches.
…, then takes their softmax steps in order A warp walked its chunk one key at a time: the key's load, its warp sum, then the value's load, each waiting on the last, so a 128-key chunk was bound by load latency. The keys' rows and values now load eight at a time and their dots reduce together; the running max, sum and output still take each key in turn with the same operations, so every value is unchanged. On an RTX 5090 a 16-row round drops from 16.24 to 15.85 ms with bf16 rows and from 16.49 to 16.03 ms with e4m3 rows; four keys at a time gave 15.95, sixteen 15.91.
…a long match or out of the prompt early Every appended copy's rows were priced from cfg.copy_rate's 0.94, a prior worth sixteen rows that a stream's one or two copies never moved, so a lone window kept growing past 16 rows for copies that did not land. Traced over the 134-prompt set on Qwen3.8-27B: a copy after a context match of 20 or more tokens landed 85% of its rows (code's), one copied out of the prompt in the reply's first 64 tokens 89% on code (restating the asked-for function), and the rest 0 to 13%; outside code, rows past the 16th landed 11 of 777 priced. Those two kinds keep the prior; the others start from 0.08 with a history of their own, which trusted copies teach too (a stream sees too few of the others to learn them alone). The proposer reports where its last match's continuation starts (Proposer.lastEnd). Decode tok/s, three runs each, every reply unchanged: code 810 / 819 / 813 before and 803 / 817 / 813 after, gsm8k 525 to 533-534, mgsm 331-332 to 335, mmlu 354 to 357, rounds 16.06 to 15.92-15.95 ms.
…m's tokens leave them The drafter's logits came from the whole 248,320-row head, 450 us a round, where DFlash2's vocabulary (the Metal drafter's and the qwen27 engine's) is the head's first 98,304 rows and its last 288: 182 us, the first span a view of the head's tiles and the tail projected from the tile at 248,000. Drafting only those spans costs multilingual replies, whose tokens sit between them (MGSM 332 to 230 tok/s), so a stream drafts over the spans until its prompt or reply holds a token outside them, then over the whole head. A lane hold now carries the stream's tokens (lanes.drafter.Hold.tokens), and the drafter reads each stream's new ones once. Verification decides every token, so replies are unchanged: on the 134-prompt set all 364 to 367 tok/s (code 813-817 to 831, mgsm 335 kept), a round 15.92 to 15.73 ms.
… row for the sort and the race tf_draw_rows took each sampled row on one 1024-thread block passing over the whole vocabulary several times: 195 us a round, 16 blocks on a 170-SM GPU. Both of gpu_sample's candidate paths (a window near the max, or the 1,024 largest) reach the same thing for a top_k up to 1,024: the row's top_k by value then id, which its tail sums and races. So for a top_k up to 32 each 8192-entry slice now takes its own top_k (tf_draw_slice_top), and a block a row sorts the slices' candidates and runs the same tail (tf_draw_rows_finish); other rules keep the one-block draw. Every pick is unchanged: the community-v1 fixtures' sampled replies and the 134-prompt set sampled at 0.7 are byte-identical, and a sampled round drops from 15.59 to 15.39 ms (code at temperature 1 410 to 415 tok/s).
…utputs in one butterfly, its slots in shared A tree window's delta-rule scan reduced each value row's k . state sum and then its output with their own warp_sum, four butterflies a step for two rows, and kept its slots in registers that every step copied through. As the qwen27 engine's tf_gdn::tree_kernel does, a step's sums and the previous step's outputs now reduce in one spread_sum (the same lane pairs as warp_sum, so every sum is unchanged), the output written a step later, and the slots are the warp's rows of shared memory. Every reply is identical; on the community-v1 fixtures a round drops from 15.43-15.49 to 15.29-15.35 ms, and on the 134 prompts all 367 to 369 tok/s. The contract's static shared sizes follow.
…in 16-byte loads tf_q35_tap took a row on one 256-thread block of scalar loads (16 blocks, 9.3 us a launch) and tf_q35_mlp_act one entry a thread (2.0 us a launch, 70 a round). The tap now runs (rows, D / 2048) blocks and the activation 2048 entries a block, both eight entries a thread through uint4 loads and stores, each entry's arithmetic as before; the activation's 32 group sums a block still run their 64 entries in order. Every reply is identical; a round on the community-v1 fixtures drops from 15.29-15.35 to 15.23-15.29 ms.
…get draw's own Gumbel noise A sampled round accepts a draft when the target's draw picks it, and that draw is a Gumbel race keyed by the stream's seed, the position and the token. The drafts ranked the drafter's candidates by its logits alone, so at a temperature the tree followed the drafter's mode while the draw followed its noise. As the qwen27 engine's draft_tree.best_first does, a sampled hold now scores each row's candidates at the stream's temperature plus 0.7 times that draw's noise (lanes.gpu_rule.uniform24 at the draft's position), both for chains and lanes (choice) and for grown trees; a lane hold carries the stream's sampling (lanes.drafter.Hold.sampling). Verification decides every token, so replies are unchanged. On the community-v1 fixtures, every reply identical: code at temperature 1 198 to 195 rounds and 420 to 430 tok/s, chat at temperature 1 346 to 322 rounds and 236 to 250 tok/s.
…e activation takes four entries a thread For a recorded window of two rows or more, gdn_pre logs every row's conv tail and tf_q35_conv_commit returns at once, yet each of the 48 linear layers launched it: one window's rounds (the lone stream's) no longer do. The MLP activation now takes four entries a thread, 1024 a block (16 group sums, each in order), twice the blocks on an RTX 5090's 170 SMs. Every reply is identical; on the community-v1 fixtures a round drops from 15.25-15.50 to 15.15-15.40 ms (code at temperature 0 362 to 364 tok/s).
…base tf_draw_rows moves into sample_rows.cuh, and graftProspect into windows.zig beside the candidates() it reads; both unchanged. The verify-rows comment takes one line.
BobClawblaw
force-pushed
the
q35-cuda-prefill
branch
from
October 21, 2026 22:04
2bd805a to
27e6f51
Compare
Contributor
Author
|
Rebased with #1079 onto 1.1.2: no conflicts in this PR's two commits. The base is #1079's new head, Checked on one RTX 5090 together with #1168 (the stack's head), on fresh servers:
|
…lines capped at 256 rounds The kept lone calibration (2c9871f) started every lone stream from the newest finished one's round times and trial deadlines. Two things in it let one long prompt slow every short reply after it until a restart: - the times carry no context length: a 253,039-token stream left 13 to 16-row rounds at 30 ms, and the short streams after it, whose 16-row round takes 15 ms, ran 10-row windows on those prices; - a losing trial (the one-row trial nearly always loses) at least doubles the spacing, and carried from stream to stream it grew without bound: 4,096 rounds after two minutes of soak, more after a long run, so no later stream re-timed the wide widths. Calibrations are now kept a context band each (under 8,192 tokens, then one a doubling, the last from 131,072 on), greedy and sampled apart, so a stream starts only from one measured at its own length and a long prompt leaves a short stream's calibration in place. A kept deadline or spacing reaches at most 256 rounds into the next stream. On one RTX 5090 (Qwen3.8-27B MLX 4-bit + DFlash2, fp8 caches, this stack's top with --prompt-precision int8): the quality suite's soak, then one greedy reply on a 252,783-token prompt, then the decode ruler's prose and code prompts. Before: 648-653 drafts proposed over 72 rounds (238 tok/s) and 379 over 41 (415 tok/s). After: 990 over 66-67 (253-257 tok/s) and 510 over 34 (495-496 tok/s), as on a fresh server. Replies are byte-identical either way.
…ule chain, the 2x2-warp prompt tile - prefill_attention_rows.cuh takes e4m3 cache rows with an fp32 scale each (F8: staged a step ahead and widened to bf16 in the slot) and head-major caches (HM: each key head's CAP rows in a run). hpattn_kernel and hpattn8_kernel instantiate them at head dim 256. pattn_kernel, tree_kernel, full_kernel and vision_kernel compile to the same SASS as before on sm_120. - gdn_prefill.cu: v0.6.6's prompt chain for the gated delta rule (two threads a value row, 32 steps staged), its strength bf16 or fp32. - qmm_prefill.cu adds the prompt tile on 2x2 warps and two stages; Affine4Kernels.half_prompt picks it.
…the delta-rule chain A prompt pass ran in 128-row chunks: each projection's GEMM launched about 80 blocks on a 170-SM GPU, attention ran the decode kernel's one-warp-a-row partials, and the delta rule stepped through rows one warp a value row. On one RTX 5090 (Qwen3.8-27B MLX 4-bit + DFlash2, fp8 KV, a 262,144-token window) a 75,616-token prompt took 144.5 s to its first token, against 26.6 s in the earlier qwen27 engine. - A prompt chunk is up to 4096 rows (Round.max_rows, the tables' and the activations' rows; the snap grid follows). Verify windows stay within most_verify, and the decode partials are sized for those rows only. - A one-part prompt chunk's attention runs hpattn(8)_kernel over the stream's own cache, and its delta rule gdn_prefill's chain, from pointers the host reads off the staged segment (prompt passes are not captured). - Prompt GEMMs take the 2x2-warp tile except on a GB10. - The drafters absorb a chunk's taps in pieces of most_verify rows, so their scratch does not grow with the chunk. The same prompt now reaches its first token in 26.9 s (56,143 tokens: 18.6 s); a needle at 40% of 60,000 tokens is found; decode rounds cost the same (16.13 to 16.20 ms). Prompt arithmetic changes with the kernels, so replies differ from 1.1.1's where its rounding decides a token; cache-on and cache-off passes still cut at the same rows.
BobClawblaw
force-pushed
the
q35-cuda-prefill
branch
from
October 22, 2026 09:29
27e6f51 to
12f8a9c
Compare
Contributor
Author
Contributor
|
Data from one GB10 (sm_121, CUDA 13.0), since this PR's numbers are from a 5090. We built on the #1078 + #1079 + #1080 stack ( On top of it we have exact follow-ups, kept local until this stack lands:
Together they take that prompt to 14.35 s (2,231 tok/s), with the FP8 tile measured at another -4.5 % separately. Token hashes are unchanged and drafted == plain, concurrent == solo and resumed == fresh hold. They touch |
ashhart
added a commit
that referenced
this pull request
Oct 24, 2026
…e axes The host staged an image chunk's t, h and w positions in three planes a max_rows apart, but the device read them a MAXR (128) apart. The h plane's row 3 was therefore read from the t plane's row 131, and every row past 128 of a long chunk was read from the wrong plane. The planes are now interleaved: a row carries its three axes together, and the device reads M[m * 3 + a]. The stride no longer depends on the row count, so the host and the device cannot disagree as the chunk's rows grow. The shared layout lives in cuda_round.zig's mropeOffset and stageImagePos, and the engine stages the chunk through cuda_engine.zig's stageImagePos. The host test stages the chunk's rows at 127, 128, 129, 4095 and 4096 rows and reads them back at the device's offsets. Before this change it failed: the h plane's row 3 held the t plane's row 1's position.
ashhart
added a commit
that referenced
this pull request
Oct 24, 2026
…ission The lead's ruling on the geometry: unsupported query/KV ratios and row counts are refused at admission, before allocation, cache settlement or queued mutation (T268 odd-group item); no silent fallback that changes the supported domain. The family supported two paths and said so only in the forward's body: the prompt tiles take an even query/KV group and a prompt chunk's rows, the decode partials take the verify rows. An odd group's long prompt fell through to the partials and was refused there, after the embedding, the projections and the rotary writes had been queued and the caches settled. The domain is now stated once, in cuda_geometry.zig's supported, and the engine refuses an unsupported geometry at admission with error.UnsupportedQwenGeometry. The domain is the one the forward already ran: the partials still take the verify rows whatever the group, so nothing that ran before is refused now. The host test admits an even group's long prompt, refuses an odd group's, and refuses a verify window past the partials' rows. Before this change the odd group's prompt chunk was admitted: the test found no error.
ashhart
added a commit
that referenced
this pull request
Oct 24, 2026
…ssion The unsupported geometry was refused at admission, but after the request's own state had been read: a full cache was reported as a full cache, and only then would the geometry have been refused. The forward's body kept a second refusal of the same rows, so the domain was stated in two places. The refusal is now the first thing the admission does, before the parts, the caches and the tokens are read, and the forward's late refusal is gone: the supported domain is stated once, in cuda_geometry.zig's supported, and refused once, at admission, before the staging, the cache settlement and any queued launch. The host test refuses an unsupported geometry for a request whose cache is full, and holds that nothing was queued and the earlier round was not settled. Before this change the full cache was reported first: the test found error.QwenContextFull.
ashhart
added a commit
that referenced
this pull request
Oct 24, 2026
The lean gate counts a comment past 120 columns and a file over 600 lines as new findings. The image planes' comment and the supported geometry's comment are brought inside those bounds, and the engine's image rows and staged planes move to the round module that owns the staged tables.
ashhart
added a commit
that referenced
this pull request
Oct 24, 2026
The lead's M54 review: the cost-open host test fails from the #1168 boundary on, because the admission refusal reads a geometry the test's probe never set. The refusal is right; the probe was wrong. The warm path's probe stands in for the target engine, whose geometry the real engine holds from the checkpoint's config (cuda_geometry.zig:248); the probe built its engine with `var engine: E.Engine = undefined` and set only `g.vocab` (cuda_warm_test.zig:56), so the guard read undefined query and KV heads. Under #1168's group bound (cuda_geometry.zig:148) that group is neither even nor in 2..16, and the one-row reservation the warm path opens (cuda_reserve.zig:16) is refused with error.UnsupportedQwenGeometry. Every served shape is inside the domain the guard states: the 27B's 24 query heads over 4 KV heads (cuda_geometry.zig:373), the 35B-A3B's 16 over 2 (cuda_geometry.zig:329), and the rest the repo carries (8/2, 32/8, 64/8, 4/2) all give an even group in 2..16. The refusal is not too strict; the probe had no geometry at all. The probe now reads the served target's query and KV heads (the 27B's 24 over 4, an even group the prompt tiles take) at cuda_warm_test.zig:58, and one case asserts the refusal still stands: a window past the partials' rows, and a group the prompt tiles cannot take, are refused at admission (qwen35_cost_open_host.zig:107). Before this change the step failed 1/3 from the #1168 boundary on (the #1080 boundary passed 3/3: the group bound arrived with #1168); after it the step passes 4/4 at every boundary.
Owner
|
Landed on main with the Qwen CUDA stack (through 6244c62), thank you @BobClawblaw. 4,096-row chunks, tensor-core attention and the rest of the long-prompt path are in. Measured on a GB10 with Qwen3.8-27B MLX 4-bit against main: cold prefill 834 to 1,281 tok/s at 2k (+54%), 612 to 1,266 at 8k (+107%), 326 to 1,165 at 32k (+258%), and earlier 203 to 1,045 at 64k (+416%). The decode sweep is +1.8 to +2.2% at 1 to 8 streams. Quality equals main's: 27 of 40 exact cases on both builds. Under our new-recipe ruling, the stack's own exactness holds: every stream equals its run alone, drafted equals plain, and resume equals fresh. - Synthema, Ash's AI Assistant. |
Owner
|
This shipped in TensorFold 1.1.4: https://github.com/ashhart/TensorFold/releases/tag/v1.1.4 Thank you @BobClawblaw for the contribution. - Synthema, Ash's AI Assistant. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #1079 (its nineteen commits come first here); the last two commits are this PR's.
Long-prompt prefill for the Qwen3.5 CUDA family. Measured on one RTX 5090 (Qwen3.8-27B MLX 4-bit + DFlash2,
--kv-cache fp8, a 262,144-token window): a 75,616-token prompt took 144.5 s to its first token on main and takes 26.9 s here (56,143 tokens: 18.6 s).A prompt pass ran in 128-row chunks: each projection's GEMM launched about 80 blocks on a 170-SM GPU, attention ran the decode kernel's one-warp-a-row partials, and the delta rule stepped through rows one warp a value row. Nsight's split for the 75,616-token prompt: attention 83 s, projections 56 s, the delta rule 3.4 s.
Kernels
prefill_attention_rows.cuhtakes e4m3 cache rows with an fp32 scale each (F8: staged a step ahead and widened to bf16 in the slot) and head-major caches (HM: each key head'sCAProws in a run).hpattn_kernelandhpattn8_kernelinstantiate them at head dim 256.pattn_kernel,tree_kernel,full_kernelandvision_kernelcompile to identical SASS on sm_120.gdn_prefill.cu: v0.6.6's prompt chain for the gated delta rule (two threads a value row, 32 steps staged), β bf16 or fp32.qmm_prefill.cuadds the prompt tile on 2x2 warps and two stages;Affine4Kernels.half_promptpicks it (off by default; the Qwen3.5 engine sets it except on a GB10).Engine
Round.max_rows, the tables' and activations' rows; the snapshot grid and cold-cut rule follow, so cache-on and cache-off passes still cut at the same rows). Verify windows stay withinmost_verify, and the decode partials are sized for those rows only. About 0.6 GiB more scratch on the 27B; a 262,144-token window still fits one stream on 32 GB.hpattn(8)_kernelover the stream's own cache, and its delta rulegdn_prefill's chain, from pointers the host reads off the staged segment (prompt passes are not captured).most_verifyrows, so their scratch does not grow with the chunk.Numerics. Prompt arithmetic changes with the kernels, so greedy replies differ from main's where its rounding decides a token (on a 24-prompt mix, 10 of 24 replies are byte-identical to main's; the earlier qwen27 CUDA engine matches main on 11 of 24 with the same kernels). Decode rounds cost the same (16.13 to 16.20 ms). A needle at 40% of a 60,000-token prompt is found; greedy replies at 75,616 tokens are identical across fresh servers.
Checks:
zig build test,test-kernel-contract,test-golden,fuzz-cortheon, the Flash Next gate test,zig fmtand the lean check pass, as do thetest-qwen35-*host steps (the cold-cut test is now written in terms ofmax_rows).