[jschmied]

Follow-ups to #1188 (prompt steps beside decoding streams), found in a review of main. Each commit has a test that fails on main and passes here. The Gemma 4 fix from the same review is a separate PR.

1. A burst of short prompts could pause decoding without bound (4c22c055b)
The short-prompt loop admits and fills until no short prompt waits. If a reply ends together with its prompt (max_tokens 1, an embedding, a cancel), it frees its lane inside the loop. Under steady arrivals of those, decoders got no round. Now, after a lane count of short prompts have begun, a round goes first. A burst that fits the lanes still goes in together, so normal traffic is unchanged. Test: 40 one-token requests beside a decoder on 2 lanes. Main fills all 40 back to back; now at most 2 in a row.

2. Short prompts waited behind an aged long prompt's debt (3dc3997d5)
pick() let a long prompt that had waited 8 choices win. The short loop then ended, and with decode time owed, the long step wasn't due either, so a new short prompt waited out the whole debt. Short prompts are now picked among themselves (same order: aged, foreground, least left). The long step picks among long prompts. A burst past the lane count yields to a waiting long prompt as it does to a decoder, so aging still holds with no decoder. Tests: share 0.01, a decoder, a 1000-row prompt and 30 one-token requests. On main one short prompt waited 63 decoder rounds; now at most 1. A second test checks that with no decoder, a long prompt beside 40 one-token arrivals still goes in before the last of them.

3. Debt survived its decoders (138e7a400)
If every decoder finished or paused, the debt stayed, and the next decoder was paid it before the long prompt's next step. The debt is now cleared whenever no stream is runnable. Test: a 3-token decoder ends after a long prompt's first step at share 0.01, then a second decoder arrives. On main it ran 50 rounds before the long prompt's next step; now 1 (its first token).

Gates: full zig build test -Dnvcc exits 0. Lean: 0 new problems against the merge base. The fake-backend equality tests are unchanged and pass. With the default share (1), scheduling changes only when lanes free up inside a burst (point 1).

[jschmied]

Metal check, since lane_host is shared. M3 Pro (36 GB), macOS 26.7.1, main 8086c205c against this branch 138e7a400:

• Full zig build test: no new failures. One core.paced_read test failed once while an 18 GB download was loading the disk. It passed in 5 reruns, and this branch doesn't touch that file.
• Two-stream smoke: tensorfold-native serve --parallel 2 --context 4096 --temperature 0, drafts on. Two prompts, solo, then two concurrent rounds.
  • Models: Qwen3.5-2B 4-bit, and Nemotron 3.5 Lightning 30B-A3B 4-bit with MTP.
  • Every concurrent reply equals its solo reply, and the replies are identical on main and this branch.
  • Timing is within noise. On Nemotron the concurrent rounds took 49 steps instead of 36 with fewer accepted drafts, so the streams did overlap.

[ashhart]

Landed on main (84e0d56..24072ec), with your three commits as author. Thank you! On a GB10 (Qwen3.8-27B CUDA) against main: tokens identical, the 1-8 stream sweep exactly flat, and prefill within 0.3% across six alternating runs. On an M3 Ultra it is flat as well, and with the rest of the batch, Flash Next MLX on CUDA is flat to +2.8%.

- Synthema, Ash's AI Assistant.