linalg/qr: regenerate timed inputs per repeat and mirror the stream rule

[SamMausberg]

Fixes #148 (gaps 2, 3, and 4; gap 1 was fixed by #150).

qr_py/eval.py and qr_v2/eval.py get the same change and stay byte-identical, as on main.

Gaps 2+3: output replay in the timed region

The timed loop reused the same input objects on every repeat, and the shapes that dominate the leaderboard geomean have a data_list of length 1. A kernel could compute outputs during untimed calls (warmup, the pre-timing validation pass, earlier repeats), cache them by id() or content, and replay them inside the timed window at near-zero cost. The recheck from #150 cannot catch this, because a replayed output is still correct for that same input. This is the capped_sleepy_cache mechanism named in the issue.

Every timed repeat now gets fresh input content:

• _timed_repeat_batch() derives per-item seeds through the existing _combine, from a shift unique per (invocation, repeat, item). % 2**63 keeps the nested result inside what manual_seed accepts. Disjointness is exact for local task seeds (verified exhaustively); with a server-combined base seed the mod can wrap, making collisions negligible-probability rather than impossible.
• Each _run_single_benchmark invocation gets its own seed_salt (benchmark: warmup 0, timed 1+idx; leaderboard: warmups 1..N, timed N+1..2N). Untimed content never reappears in a timed window, even for a cache persisted to disk across the test, benchmark, and leaderboard runs of one submission.
• Generation and cloning happen before clear_l2_cache(), outside the timed CUDA-event window, so per-repeat timing is unchanged. The warmup / pre-timing batch is bit-for-bit identical to today.

An id-keyed cache never hits (fresh objects), a content-keyed cache never hits (fresh content), and a stale replay fails the existing recheck.

Not added on purpose: an element-wise cross-check against torch.geqrf. (H, tau) is not unique (signs, blocking, inner precision), so exact matching would reject valid implementations the tolerances are meant to admit.

Stability-break floor

Found in A/B runs: the untimed per-repeat work arms the early-stop rule (err/mean < 0.001 once wall time passes 0.1s) almost immediately, and on a large shape it fired on a 3-sample estimate (runs: 44 -> 3). The break now needs at least 10 timed samples; the same shape then takes 37. The mean*runs and 120s budget caps are untouched, so worst-case wall time is bounded as before.

Behavior changes

• Kernels that clobber their input in-place now pass benchmark/leaderboard (each repeat gets a fresh buffer). On main they failed at repeat 2. Test mode always handed kernels a clone, so this aligns the modes.
• mixed benchmark lines draw a fresh profile assignment each repeat, so their mean ranks a distribution of instances instead of one fixed instance.

Gap 4: the stream rule, mirrored locally

KernelBot rejects any submission containing the substring stream (case-insensitive, comments included) at intake (api_utils.py and leaderboard_cog.py in discord-cluster-manager). Nothing checked this locally, so stream-using kernels passed every mode and only failed on their first remote submission. eval.py now runs the same substring test before dispatching any mode, emits a report entry plus the reason on stderr, and exits 112. Only the submission file is scanned, matching the remote rule.

Validation

RTX 5070 Ti, torch 2.8.0+cu128; this checks harness semantics, not B200 performance. Runs staged the real problem files exactly as KernelBot lays them out, on a true count=1 shape (batch 256, n 512), benchmark mode:

Submission	main	patched
torch.geqrf reference	passes, mean 448.0 ms/input	passes, mean 447.0 ms/input
shape-keyed output cache	passes, mean 5.3 us/input	fails first timed recheck
content-keyed output cache	passes, mean 203 us/input	passes, mean 448.8 ms/input (advantage gone)
id()-keyed output cache	passes, mean 3.0 ms/input	fails timed recheck (stale replay caught)
fp64 inner math (valid factors, differ element-wise from fp32 geqrf)	passes	passes
in-place input clobbering (correct output)	fails at repeat 2	passes
side-stream kernel (correct, synchronized)	passes locally, rejected at remote intake	fails every local mode with the stream-rule message

Reference means agree to 0.2% between harnesses; leaderboard mode passes; test mode passes on the real task.yml shapes including mixed and the stress cases. Seed uniqueness was verified exhaustively, and timed batches are deterministic per (salt, repeat) with the warmup batch unchanged.

Not measured: B200 wall clock against the task timeouts. Here the three-shape reference benchmark was within 5% of main (237.6s vs 228.2s), but a B200 number would be worth taking before merge.

Follow-ups, out of scope: eigh_py and cholesky_py share the same timed loop and lack the stream gate, and per Bryce's comment on the issue the gate is worth propagating to the other problems.

Made in combination with Claude Fable 5.