[jschmied]

What this changes

Flash Next's PLE n-gram gather (Ngram.rows in families/flashnext/cuda_ple.zig) spawned min(128, items / 4) threads per call and joined them. The gather itself is a few memcpys from the mapped table, so most of a call was thread start-up: about 65-80 µs per thread, 150 µs for one decode token, 1 ms for a 4-token batch (4 streams, or a drafted verify) and 10-11 ms per prompt chunk.

The table now keeps a pool for its life (cuda_ple_pool.zig, 139 lines with a unit test):

• workers are spawned on first need, up to 127, and park on a futex between calls; deinit wakes and joins them;
• a call's parts are the same [lo, hi) ranges as before, claimed from one counter by the caller and the workers. Each claim that leaves parts unclaimed wakes one more worker, so quick parts stay mostly on the caller and slow (faulting) parts fan out;
• calls are serialised by a mutex, and the first failed part's error is returned, in part order, as before.

Only which thread copies a range changes, so the output bytes are identical by construction. A table without a pool (for example a path that builds Ngram some other way) gathers on the caller's thread.

Branch fn-ple-pool @ e4062e6a, on main @ bd264368a.

Receipt

• Environment. One GB10 (sm_121), CUDA 13.0, Zig 0.17.0. nvidia/Qwen3.8-Flash-Next-FP4-Local @ b9ce18a6 (its tensors/ index loads since Checkpoints: an index may name a shard in a folder inside the model folder; shard edits keep plain names #1179).
• rows() on the real table, p50, alternating old / new / old / new, output hashes identical at every size:

items	main	this PR
16 (one decode token)	165 / 163 µs	18 / 22 µs
64 (4 tokens)	1,020 / 971 µs	31 / 32 µs
4,096	11.4 / 10.4 ms	0.95 / 0.97 ms
8,192	11.5 / 10.7 ms	1.66 / 1.66 ms

• Served, a fresh server per arm, main / this PR / main / this PR with --no-drafts, then one MTP pair:

	main	this PR	change
c=1 ms/step	42.00 / 42.14	41.90 / 41.74	-0.1 / -0.4 (noise-sized)
c=4 ms/step	61.11 / 61.87	60.09 / 60.84	-1.02 / -1.03
TTFT 8k	6.365 / 6.481 s	6.335 / 6.432 s	-0.03 / -0.05 s
MTP c=1 ms/round (one run)	60.52	59.44	-1.08
MTP c=4 ms/round (one run)	223.6	218.9	-4.7 (-2.1 %)

The c=4 gain has the same size and sign in both rounds. The microbench accounts for about 0.65 ms of it (4 lanes x 4 calls a round); the remaining ~0.35 ms is not explained. Prompt chunks mostly hide the old cost, because the batch path gathers on a side thread while the layers before the PLE layer run.

• Exactness. Token hashes for greedy c=1 and c=4, seeded c=1 and c=4, and both turns of a resumed prompt-cache run: identical across all six servers, the MTP ones included.
• Tests. Full zig build test -Dnvcc=... exit 0. lean_check 0 new. zig fmt clean.
• Contract. No kernel change.

Not run, and why

• Metal: cuda_ple.zig belongs to the CUDA family only, so there was nothing to run there.
• The one-thread gather cuda_batch.zig starts per pass (~28 µs) could use the same pool; left for a separate change.

Overlap

#1113 sets the thread count for its EXL3 table in the same function and builds that table without going through Ngram.init, so a rebase there needs a hand-merge; its table then simply gathers on the caller's thread, as before.

[ashhart]

Landed on main as e2cfdae with a maintainer fix-up (8e9ab68) covering the pooled gather's bytes and worker ownership, thank you @jschmied. The Flash Next per-layer embedding (PLE) gather now uses a parked worker pool instead of spawning a thread per call. It came through the union campaign's Flash Next cells with identical tokens, and the host suites pass on current main.

- Synthema, Ash's AI Assistant.

[ashhart]

This shipped in TensorFold 1.1.4: https://github.com/ashhart/TensorFold/releases/tag/v1.1.4

Thank you @jschmied for the contribution.

- Synthema, Ash's AI Assistant.