Repository navigation
GB10 sm_121 latency kit versus NAK - #254
Conversation
Dependent-chain kernels, a NAK diff, and a Spark runner that holds flock /tmp/aien-gb10.lock only around the launch (timeout 600). Measured cells stay empty until that run; the host self-test does not invent them. Co-authored-by: aien-spark <[email protected]>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The fixed-latency chains read a destination armed to a sentinel the producer does not read, so a stale value fails the oracle. Stall-1 SASS is checked for that pair with no scoreboard and no yield. Cycle medians at stall 1 are recorded even when every stall matches. GB10 open still depends on the 0xa04 admission; this does not add it. Co-authored-by: aien-spark <[email protected]>
sm121dis of the overhead kernel showed IADD of the counter with stall 1 (control 0x000fc200) immediately before ISETP, and @!P0 BRA back to the body. The compare then samples the previous counter, so the stored trip count misses the overhead check and every exact oracle fails through stall 15. HMMA has no loop. On the predicate chain, stall 1 leaves the previous true P0 for @p0 BRA, so the poison move exits on the first pass and the clocks cover that exit. The host simulator hides GPR writes for 5 cycles and predicate writes for 14. Overhead must store the trip count, the top of the sweep must match the fresh oracle, stall 1 must match the stale oracle, and putting stall 1 back on the increment must fail the counter check. Co-authored-by: aien-spark <[email protected]>
The launch is cb5bd70, the merge of 03cdcc5 and PR #251 6a93cb5. raw.json, the filled table, and sm121dis --pairs of each threshold and the stall under it are the chip output. UNITS.md compares that stall field with NAK's deps.delay at Mesa 01bab930. Co-authored-by: aien-spark <[email protected]>
Each lane stores the two clocks, four result words, and its lane id. HMMA's four words are FADD of d0..d3 against zero, compared with the host dot of sixteen bf16 ones, not with the stall-36 launch. The host simulator requires the same match on all 32 lanes. This has not been launched. Co-authored-by: aien-spark <[email protected]>
c94c2c9 is the merge of 4f3a4d0 and PR #251. stall_required matches the earlier receipt. raw.json still has one lane-0 word and a null reference, so the file does not itself show the 32-lane or d0..d3 check. Co-authored-by: aien-spark <[email protected]>
raw.json now has the four oracle words, lanes_matched_min, a lane fail mask, and the first mismatching run, lane, and word. run_spark.sh writes RECEIPT.txt and no longer prints a 90 s bound beside the binary's 71 s. Co-authored-by: aien-spark <[email protected]>
fb91560 is 7ec0184 merged with PR #251. Passing stalls have lanes_matched_min 32. Every recorded first_mismatch is word 2, so the file does not by itself show a later word failing. Co-authored-by: aien-spark <[email protected]>
word_mismatch_max counts lanes that miss words 2..5. The receipt uses PCI 10de:2e12 when proc Model is Unknown, which is the doctor's GB10 row. Co-authored-by: aien-spark <[email protected]>
The path passed to fopen is the case id and a stall from that table. The scanned token is only compared. Emit and the stall sweep are unchanged. Co-authored-by: aien-spark <[email protected]>
aien-spark
left a comment
aien-spark
left a comment
There was a problem hiding this comment.
draft, so this is an early look. reviewed at 4a5184b (head moved from 0c81899 mid-review; the case-table load looks like it should clear the codeql path alert, but analyze (c-cpp) and host tests were still running when i looked).
- none of the three evidence dirs are for this head. -oracle is fb91560, -allanes is c94c2c9, the base one is cb5bd70, and all of them merge in #251. #246 already landed the device-open prereq, so rebase on main, drop the #251 dependency (body line 35 still says it needs it) and re-run on the chip at the head.
make testandmake test-native-artifactare NOT RUN. no fresh-clone receipt yet, which REVIEWING §1 asks for.- -oracle raw.json sha matches its RECEIPT.txt. -allanes only has RECEIPT.md, no RECEIPT.txt.
- benchmarks-mv has no baseline to compare these against.
- these are minimums from one warp on an idle SM, stall-only, and hmma has no scoreboard. don't feed them into K_FIXED / fixed_pair_class without a run under contention.
fix first (early, draft)
| if [ -f "$out/raw.json" ]; then | ||
| python3 tools/latmeas/fill_template.py --results "$out/raw.json" --md "$out/sm121_vs_NAK.md" | ||
| fi | ||
| write_receipt "$out" |
There was a problem hiding this comment.
write_receipt doesn't record the launch command or its rc, and REVIEWING §2.1 wants both. git diff --quiet also skips untracked files, so a dirty tree can still come out "clean".
There was a problem hiding this comment.
The receipt now records the launch command and its exit code. write_receipt prints launch_command and launch_rc for the exact flock /tmp/aien-gb10.lock timeout 600 ./build/gpu_latmeas_test --launch ... line. Dirty detection uses git status --porcelain, so an untracked file marks the tree dirty. The --receipt-self-test path checks both the command and rc, and a temporary git repo with an untracked file. This is 23f2663.
| | ALU -> ALU (GPR RAW) | 4 + 1 = 5 | latencies/sm100/reg_raw.csv:2 row alu column alu = 4; sm120_instr_latencies.rs:457 GPR raw adds max(adjust, 1); IAdd3 is Alu at line 44 | 4 | -1 | 11 runs; stall spread 0; cycles at that stall median 2541 spread 0; stall-1 cycles median 2477 spread 0; loop overhead median 1261 | tests/gpu_latmeas_test.c (alu_alu) | | ||
| | FMA -> FMA (GPR RAW) | 4 + 1 = 5 | latencies/sm100/reg_raw.csv:5 row fma column fma = 4; sm120_instr_latencies.rs:59 FFma is Fma; line 457 adds 1 | 4 | -1 | 11 runs; stall spread 0; cycles at that stall median 2541 spread 0; stall-1 cycles median 2477 spread 0; loop overhead median 1261 | tests/gpu_latmeas_test.c (fma_fma) | | ||
| | Mixed ALU <-> FMA (GPR RAW) | 5 + 1 = 6 | latencies/sm100/reg_raw.csv:2 row alu column fma = 5 and line 5 row fma column alu = 5; sm120_instr_latencies.rs:457 adds 1 | 5 | -1 | 11 runs; stall spread 0; cycles at that stall median 4362 spread 0; stall-1 cycles median 4170 spread 0; loop overhead median 1261 | tests/gpu_latmeas_test.c (mixed) | | ||
| | Predicate write -> guard | 13 + 1 = 14 | latencies/sm100/pred_raw.csv:11 row redirected_fp64 column dualalu = 13; sm120_instr_latencies.rs:474 predicate raw(write, None) uses reader RedirectedFp64 and adds 1; ISetP is Dualalu at line 49 | 13 | -1 | 11 runs; stall spread 0; cycles at that stall median 2714 spread 0; stall-1 cycles median 70 spread 0; loop overhead median 1261 | tests/gpu_latmeas_test.c (pred) | |
There was a problem hiding this comment.
pred at stall 1 reads 70 cycles, which is under the 1261 loop overhead. at that stall the guard reads stale and the loop just exits, so this column is measuring a control-flow change, not latency. it's also labeled "stall-1", which reads like S-1, but it's cyc[..][0] (stall=1). either drop it or label it and flag failing stalls.
There was a problem hiding this comment.
This is not a bad stall measurement. cycle_median_stall1 70 is the raw clock at programmed stall 1 (cyc[][][0], tests/gpu_latmeas_test.c lines 1965 and 1987). It is not stall 13 minus one, and nothing subtracts overhead_cycle_median (line 1995). At stall 1 the predicate guard still sees the previous true P0, the poison branch leaves the loop, and the clocks cover that exit, so 70 is under the 1261-cycle empty loop. The pass bit is the oracle sweep: stall_required 13 is reported_threshold (line 1223) of pass[][] set at line 1884 when every lane matches. The clock at that stall is cycle_median 2714 (lines 1946 and 1981), also raw, and it is above the overhead. Neither clock decides the stall. evidence/LATMEAS-GB10-20261101-oracle/raw.json pred is stall_required 13, cycle_median 2714, cycle_median_stall1 70, overhead_cycle_median 1261. The table label is now "programmed stall 1 cycles", and the note under the table cites those lines. 23f2663.
| LCHECK(build(&CASES[6], 1, 0, 0, code, sizeof code, &len, &gpr) == 0, "hmma stall 1 encode"); | ||
| LCHECK(lat_sim(lane0) == 0, "hmma stall 1 sim"); | ||
| LCHECK(lanes_report(g_sim_mem, o, &m) == 0 && m.word >= 2 && m.word <= 5, "hmma stall 1 fails a destination word"); | ||
| char tmpl[] = "/tmp/latmeas-load-XXXXXX"; |
There was a problem hiding this comment.
mkdtemp under /tmp and nothing removes it. every self-test run leaves a full kernel dump plus manifest behind. unlink the files and rmdir at the end.
There was a problem hiding this comment.
The host self-test now unlinks every file in the mkdtemp directory and rmdirs it (tests/gpu_latmeas_test.c, the load-check block). 23f2663.
Brings in the GB10 admission from #246 (dba5821) so nvrm_open accepts SM 0xa04 on PCI 10de:2e12. Kernel bytes and stall logic are untouched. Co-authored-by: aien-spark <[email protected]>
The receipt prints the exact flock launch and its exit code. git status --porcelain marks a tree dirty when it has untracked files. Programmed stall 1 cycles stay in the table as a raw clock, not as the pass bit: the stall is the oracle sweep, and neither clock column subtracts the loop overhead. The host self-test removes the kernel dump it creates. Co-authored-by: aien-spark <[email protected]>
cursor
Bot
changed the title
|
Fresh-clone host gates for This comment is the receipt. It is not committed on the branch, so the head stays
Host results are not hardware qualification. |
Signed-off-by: aien-spark <[email protected]>
Signed-off-by: aien-spark <[email protected]> # Conflicts: # Makefile
Purpose and scope
Host-built latency kit for fixed stalls on the GB10. Kernels are CUDA++ machine code.
The headline chip receipt is
evidence/LATMEAS-GB10-20261101-oracle/.sm121_vs_NAK.mdin that directory is the table filled from thatraw.json. It was measured atfb91560, the merge of7ec0184and PR #2516a93cb5, on 2026-11-01T18:47:06Z.stall_requiredis 4, 4, 5, 13, 4, 3, 20, spread 0. At each passing stall and above,lanes_matched_minis 32.Head for the next chip run is
23f266313e95f8bae2ce8880e914537685b61bf1. That commit mergesorigin/main, so SM 0xa04 admission comes from #246 (dba5821) and this PR no longer depends on #251 to open the device.23f2663also records the launch command and its exit code, treats untracked files as a dirty tree, removes the host self-test's kernel dump, and labels programmed stall 1 cycles. It does not change emitted kernel bytes or the stall sweep.The predicate row's 70 cycles are
cycle_median_stall1: the raw clock at programmed stall 1 (tests/gpu_latmeas_test.clines 1965 and 1987), not stall 13 minus one, and not overhead-subtracted. At stall 1 the guard still sees the previous trueP0, the poison branch leaves, and the clocks cover that exit, so 70 is below the 1261-cycle empty loop. The pass criterion is the oracle sweep (reported_threshold, line 1223, frompass[][]at line 1884). The clock at stall 13 iscycle_median2714 (lines 1946 and 1981), also raw. Neither clock decides the stall.raw.jsonforpredmatches those four numbers.Command
On the Spark, at
23f2663:Build and
make doctorstay outside the lock. Only the launch is insideflock /tmp/aien-gb10.lock, wrapped intimeout 600. The binary prints the bound: 1408 launches, 71 s upper bound (50 ms each). The receipt records that command and its exit code.Evidence on this commit
The oracle receipt is a receipt for
fb91560, not for23f2663. Chip gates for23f2663are NOT RUN until that rerun. Host gates for23f2663are the fresh-clone comment: #254 (comment) . That comment is not a commit, so the chip SHA stays23f2663.make test— PASS, 73 targets, exit 0. Fresh clone of23f2663.make test-native-artifact— PASS, 47 checks, exit 0.make zero-cuda-gate— PASS, exit 0.build/hello_gpu --sim— PASS, exit 0.make doctor— NOT READY, exit 2. Expected on this VM (no NVIDIA module, no/dev/nvidia*).23f2663. PASS forfb91560only, inevidence/LATMEAS-GB10-20261101-oracle/.What merge still needs
Draft. Ruleset
main-verified-promotiondoes not require checks.CODEOWNERS.proposedis inactive.docs/REVIEWING.mdsection 1: evidence for the reviewed head, a fresh-clone host-gate receipt, approval by someone other than the author,gh pr merge --match-head-commit.Sweep re-check (2026-11-04, DRA-130)
CI: only drift fails (https://github.com/aien-dev/CUDApp/actions/runs/41910517716/job/126856525966). The #251 dependency is gone (see the head paragraph). GB10 at this head: NOT_RUN.
The drift FAIL (45 of ~920 GPU paths vs omega-mv d3d1b1bc7) is inherited from main. It clears when #266 merges.