Skip to content

Mirror omega b5ab3fe GPU drift and record the rest - #203

Merged
aien-spark merged 5 commits into
mainfrom
cursor/drift-omega-b5ab3fe-a054
Oct 31, 2026
Merged

aien-spark merged 5 commits into
mainfrom
cursor/drift-omega-b5ab3fe-a054

Conversation

@drakestapleton

@drakestapleton drakestapleton commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Evidence on head d2c83a1

  • tools/extract/drift.sh --self-test <omega-mv at 4376c2112d7469b6908c6403720c9001de4622b5> 4376c2112d7469b6908c6403720c9001de4622b5, rerun on this head (aarch64 host):
    • control 1: red as required (Makefile)
    • control 2: red as required (src/omega_gpu_attention_api.c edited in a scratch Omega commit)
    • DRIFT-GATE: PASS (783 GPU paths, omega 4376c2112d74, none moved away from CUDA++ since the baseline)
    • SELF-TEST: PASS (both controls red, real check green)
  • make test-gpu-decode-graph test-gpu-prefill-host check-own30b-decode-profile test-gpu-fp8-slots test-gpu-fp8gemm passed at 02538d3 (host and simulator only). d2c83a1 adds one baseline row and crumbs, no code.
  • crumb verify .: OK: 33 crumbs current on this head.
  • GB10 hardware tests. NOT RUN. No GPU window; /dev/nvidia* was not opened.

What this PR does now

Main already holds the earlier omega b5ab3fe sync (#207). This branch merged main, took main's side for every file of the earlier b5ab3fe commits, and now carries only the sync to omega-mv 4376c2112d7469b6908c6403720c9001de4622b5 (train r3 omega-mv#979, omega#456).

Before this branch the drift gate failed on main with 4 of 782 GPU paths changed in omega 4376c2112d74 and not in CUDA++: Makefile, src/omega_gpu_decode_graph.c, src/omega_gpu_fp8_api.c, tests/gpu_fp8gemm_test.c. Carried from omega#456:

  • the decode graph per-token timeline never writes past stamp[]; new host test tests/test_omega_gpu_decode_graph_timeline.c (fake platform, no device) and test-gpu-decode-graph-timeline with 2 mutants, run by test-gpu-decode-graph;
  • fp8 tensor upload fills and syncs under the session lock, and omega_gpu_fp8_test_set_no_fold writes under the lock;
  • gpu_fp8gemm_test emu_row uses a static buffer bounded by k;
  • test-gpu-prefill-host (8 prefill tests, host model and simulator) and check-own30b-decode-profile with tools/own30b_decode_profile.c.

CUDA++ code that was ahead stays: the fp8 upload owner generation and the test_out_open 0600 receipt in gpu_fp8gemm_test. tools/extract/drift-baseline.tsv records the 4376c21 blobs and names those paths. Not carried: the require_physics_pin.sh line in omega's chip runners (no such runners here).

Drift row: d2c83a1 adds omega's blob for tests/test_omega_gpu_decode_graph_timeline.c to the baseline, so the new mirrored test is tracked by the gate.

Crumbs: crumb compile . with crumb tool d0f0b95 also rewrote om/moe/.crumb and tools/extract/.crumb, which were STALE on main; only generated fields changed.

Removed from the earlier head 063a0c2

  • A stricter dagsim mutant check (the assertion name had to appear on a FAIL line). That is a CUDApp gate change without a Drake decision, so it is not in this PR.
  • The omega_runtime_api.internal abort_token row: main declares that function in the manifest as EXPERIMENTAL.

Known limitations

NOT_RUN on GB10: decode graph timeline and fp8 upload locking on the chip; all claims above are host and simulator only.

Grokbot

…d the rest.

Eighteen GPU paths are byte copies of omega-mv b5ab3fe94304: the IMAD.HI even-Rc model, MUFU FTZ flush, dagsim predict refusal and lifetime order, decode-graph abort, and MAX_POS 131072. Thirty-one paths stay as they are. The drift baseline records omega's blob for each, with a one-line reason. Two dagsim mutant scripts are copied and then point at src/sha256.c, because this tree has no src/omega_om_units.c.

Refs #201

— Grokbot

Co-authored-by: Drake Stapleton <[email protected]>

@drakestapleton drakestapleton left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at b3f3d5e (the head is unchanged; not a draft; base main b5746b9).

Classification: needs-changes. The PR is small and correct where it mirrors. One recorded divergence hides a real gap and should be mirrored now (tests/gpu_decode_attn_test.c, see below). Three CI jobs were also still running.

CI at this head (I didn't rely on the VM numbers)

  • drift (124335613116): success. The log says DRIFT-GATE: PASS (772 GPU paths, omega b5ab3fe94304, none moved away from CUDA++ since the baseline) and SELF-TEST: PASS (both controls red, real check green). CI's own omega clone resolved to b5ab3fe94304.
  • Passing: runtime-api, library, audit-scanner, sim, build (124335614026), build (124335612635), om-units, sbom, hello-sim, fresh-checkout, figures, Analyze (actions), Analyze (python).
  • Neutral: CodeQL.
  • Still running: host-tests (124335614249), x86-host-tests (124335613853), Analyze (c-cpp) (124335609474). Until host-tests finishes, the author's HOST TESTS PASS (58 targets) is a VM receipt only.
  • Fake-green: none found. No workflow file and no tools/extract/drift.sh change. The self-test still requires both controls red. All 375 data rows in drift-baseline.tsv carry omega's blob at b5ab3fe exactly (0 mismatches), so a recorded path goes red again as soon as omega changes it. Nothing is permanently allowlisted.

Exact mirrors: 18/18 byte-identical

I compared git blob SHAs with omega-mv b5ab3fe943048ac896b368a687a38fdc1dea4077. All 18 listed paths match, and so do the 7 companions: mk/omega-mufu-ftz.mk, tests/test_omega_gpu_decode_graph_abort.c and the 5 tests/dagsim/fixtures/*.oev. src/omega_cubin.c (frozen, ADR 0044) is untouched, and no backend// path is touched.

Recorded divergences: 31 of the 49, plus 2 extras

The count adds up: 18 mirrored + 31 recorded = 49. The two extras (tests/dagsim/mutant_lifetime.sh, tests/dagsim/mutant_predict.sh) are recorded at drift-baseline.tsv:345 and :347. Each recorded path has a # Divergence: reason on the line above its row, and CUDApp's blob does differ from omega's for every one.

My verdicts, by drift-baseline.tsv row:

Justified: CUDA++ build, docs, ABI or licence

  • :7 Makefile
  • :9 docs/runtime/API.md, :11 PORTABILITY.md, :13 omega_runtime_api.manifest, :15 portability_table.tsv (0.6.0 vs omega 0.1)
  • :158 mk/om-encoder.mk, :161 mk/omega-gpu-engine.mk, :164 mk/runtime-r1.mk, :166 mk/runtime-r3.mk (SPDX only)
  • :316 src/omega_runtime_api.h (0.6.0)
  • :406 tests/test_omega_codegen_freestanding.c (licence header only)

Justified: not imported (persistent device omega #575 / hazard checker / FP8 slot pool), with ADR 0043 Decision 0 relevance

  • Persistent device: :210 src/omega_blackwell_engine.c, :212 src/omega_blackwell_engine.h, :404 tests/test_omega_blackwell_engine.c, :351 tests/fake_m16_native.c, :353 tests/fake_m16_native.h. Not importing persistence is consistent with the Decision 0 gate. Making it permanent needs an Architecture ruling.
  • Hazard checker: :226 src/omega_bw_hazard.c, :228 src/omega_bw_hazard.h, :327 tests/bw_hazard_test.c. Its fixtures quote an nvcc listing, which is a provenance concern.
  • FP8 slot pool: :254 src/omega_gpu_fp8_api.c, :257 src/omega_gpu_fp8_internal.h, :260 src/omega_gpu_fp8_slots.c, :262 src/omega_gpu_fp8_slots.h, :367 tests/gpu_fp8_slots_test.c. It depends on src/omega_fp8_slot_table.c, which is outside the GPU map.

Justified but needs an Architecture ruling

  • :169 src/omega_blackwell_codegen.c. Besides the header and cudapp_native.h differences, I confirmed a real encoding fork: CUDApp emits LDG.E.MMIO.GPU with w[2]=0x0c1f1900, and omega now emits LDG.E.EF with 0x0c0e1900. Keeping CUDApp's word is right without chip evidence, because both are NOT_RUN on GB10. It is an instruction-encoding decision, though, and should go to Architecture. It isn't omega_cubin.c and isn't under backend//.

Acceptable deferral (comment-only, or a test-reference style difference)

  • :303 src/omega_gpu_session.h (omega's change is comments only)
  • :355 tests/gpu_attention_test.c, :384 tests/gpu_pfgemm_test.c, :388 tests/gpu_prefill_attn_test.c, :391 tests/gpu_qgemm_test.c (MUFU helper routing). These are reasonable to defer now that tests/bw_mufu_ftz.h is mirrored, but they should be follow-ups.

Deferred mirror: should be mirrored now

  • :361 tests/gpu_decode_attn_test.c. This PR mirrors src/omega_gpu_decode_attn_api.h, which raises MAX_POS to 131072, but keeps the test sampling % 65536u. The host has no coverage for the doubled cap. Omega's host-only extension costs nothing, so please mirror it (or add equivalent host checks up to 131072). Chip points above 65536 stay NOT_RUN.

Extras (justified): :345 and :347. They link src/sha256.c because this tree has no omega_om_units.c.

Makefile (hand-edited)

  • The dagsim unit link adds -Wl,--wrap=rx_cm_predict, matching omega.
  • test-dagsim runs both mutant scripts.
  • test-mufu-ftz is on HOST_TESTS.
  • test-gpu-decode-graph now depends on the abort test plus 5 sed mutants. Each mutant fails if its pattern isn't found, if it survives, or if it runs with zero FAIL lines. grep -c … || true only feeds that count, so it isn't fake-green.

Companion scripts

mutant_lifetime.sh and mutant_predict.sh both use set -eu and check that the mutation applied. They require a nonzero exit plus the named assertion in the output.

Nit: they only grep -q the assertion name, so a crash that happened to print that name in a PASS line would count as a kill. Matching a FAIL-prefixed line would be stricter.

API surface

omega_gpu_decode_graph_abort_token is added to docs/runtime/omega_runtime_api.internal:130 as INTERNAL. src/omega_runtime_api.h is not changed by this PR, so it has no version bump and the public surface stays 0.6.0. The runtime-api job passed.

Chip claims

I found no GB10 or hardware claim without evidence. The PR body marks IMAD.HI even-Rc pairs, MUFU flush, decode-graph abort, contexts above 65536, and the doctor result as NOT_RUN or host-only. The Makefile comment at the #492 note says chip probe NOT_RUN.

— Grokbot

tests/gpu_decode_attn_test.c is now a byte copy of omega-mv b5ab3fe94304, so the host checks positions through 131071. The drift baseline no longer records that path. src/omega_blackwell_codegen.c stays on LDG.E.MMIO.GPU 0x0c1f1900. Neither that word nor omega's LDG.E.EF 0x0c0e1900 has a NAK citation or a GB10 test, so under ADR 0043 provenance neither wins yet. CUDApp wrote 0x0c1f1900 in 3c16328 after the shared import; omega still has 0x0c0e1900 from 11b6f474. The dagsim mutant scripts now require the assertion name on a FAIL line.

Refs #201

— Grokbot

Co-authored-by: Drake Stapleton <[email protected]>
Comment thread tests/gpu_decode_attn_test.c Fixed
Comment thread tests/gpu_decode_attn_test.c Fixed

@drakestapleton drakestapleton left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 063a0c2. The head matches what was requested and the PR isn't a draft. Compared with my previous review head b3f3d5e, this push is one commit touching 4 files.

Classification: ready-to-merge on content, once host-tests goes green. host-tests (124339970655) was still running at 5:20 PM CT. Separately, the failing CodeQL check is the 2 expected alerts in the mirrored test (below), which still need to be dismissed. The must-fix from my last review is resolved.

1. tests/gpu_decode_attn_test.c

  • Now a byte copy of omega. Its blob is ff35e2a039efa661a056ab6a289e47a9b43cc73b, the same blob the baseline recorded for omega-mv b5ab3fe.
  • Recorded row removed. The # Divergence comment and the row (formerly drift-baseline.tsv:360-361) are both gone.
  • The 131072 cap is now covered on the host:
    • fixed positions now include 65536, 99999 and 131071;
    • the random samples are taken % OMEGA_GPU_DATTN_MAX_POS;
    • a new check refuses OMEGA_GPU_DATTN_MAX_POS + 256 before any device work.
  • Chip points 65536, 100000 and 131072 were added to the default chip mode. These are NOT_RUN: no receipt is claimed.

2. LDG word divergence in src/omega_blackwell_codegen.c (drift-baseline.tsv:168)

The comment now records the encoder word as a CUDApp edit: CUDApp replaced omega's 0x0c0e1900 with LDG.E.MMIO.GPU 0x0c1f1900 in 3c16328, after the shared import.

  • 3c16328 checked: it exists in this repo and edits this file: "Honor the LDG MMIO GPU IR contract…", 2026-10-28, 1:04 PM CT.
  • No NAK citation.
  • No GB10 claim. The comment says neither word has a NAK citation or a GB10 test, so neither wins under ADR 0043 provenance. That's correct, and it keeps the question open for an Architecture ruling.
  • The file and its baseline blob are unchanged. No frozen-stack path (omega_cubin.c, backend//) is touched.

3. Mutant scripts

  • tests/dagsim/mutant_lifetime.sh:33 now matches ^ FAIL .*inverted_lifetime_masks_overflow.
  • tests/dagsim/mutant_predict.sh:35 and :40 match ^ FAIL .*predict_unsupported_despite_known and ^ FAIL .*predict_refused_despite_known.
  • The regex fits the real output. dagsim_unit.c's CHECK writes " FAIL %s:%d " and then the message, on one line to stderr. The messages start with those assertion names (dagsim_unit.c:458, 465, 575), and the scripts capture 2>&1.
  • No fake-green. The scripts still need a nonzero exit, still use set -eu, and still check that the mutation applied. A PASS or summary line can no longer count as a kill.

4. What changed since b3f3d5e (nothing else is new)

  • tests/gpu_decode_attn_test.c (+27/−14): the full omega mirror described above.
  • tools/extract/drift-baseline.tsv (+1/−3): the reworded LDG comment, and the removed decode-attention divergence.
  • tests/dagsim/mutant_lifetime.sh (+1/−1) and tests/dagsim/mutant_predict.sh (+2/−2): the FAIL-line anchors.

There are no Makefile, workflow, drift.sh, src/ or API changes.

CodeQL (124340161204): failure, 2 new high-severity alerts. These are the expected pair in the mirrored test, where fopen creates a file with default 0666 permissions:

  • line 337, fopen(out_path, "w") in chip_run, visible in the diff;
  • line 435.

They come with the byte mirror and are being dismissed as "used in tests", so I'm not counting them as must-fix. The check reports exactly 2 annotations and no others.

5. Drift baseline

19 mirrored + 30 recorded = 49, and the 2 mutant-script rows are still recorded separately. drift (124339970752) passed at 5:11 PM CT.

6. Checks at this head (CT)

  • Running: host-tests 124339970655 (started 5:11 PM).
  • Failure: CodeQL 124340161204, the 2 alerts above.
  • Passing: x86-host-tests 124339970723 (finished 5:16 PM), drift 124339970752, build 124339971023, build 124339970583, runtime-api 124339970895, library 124339970449, sim 124339970386, om-units 124339970927, sbom 124339970943, hello-sim 124339971047, fresh-checkout 124339970759, figures 124339971011, audit-scanner 124339970836, Analyze (c-cpp) 124339959499, Analyze (python) 124339959400, Analyze (actions) 124339959221.

— Grokbot

aien-spark and others added 3 commits October 30, 2026 21:10
# Conflicts:
#	Makefile
#	tests/dagsim/mutant_lifetime.sh
#	tests/dagsim/mutant_predict.sh
#	tests/gpu_decode_attn_test.c
#	tools/extract/drift-baseline.tsv
The drift gate failed on main with 4 of 782 GPU paths changed in omega
4376c2112d74 and not in CUDA++: Makefile, src/omega_gpu_decode_graph.c,
src/omega_gpu_fp8_api.c and tests/gpu_fp8gemm_test.c. This carries the
omega#456 OWN-30B follow-ups behind them:

- decode graph per-token timeline never writes past stamp[]; new host test
  tests/test_omega_gpu_decode_graph_timeline.c (fake platform, no device)
  and test-gpu-decode-graph-timeline with 2 mutants, run by
  test-gpu-decode-graph;
- fp8 tensor upload fills and syncs under the session lock, and
  omega_gpu_fp8_test_set_no_fold writes under the lock;
- gpu_fp8gemm_test emu_row uses a static buffer bounded by k;
- test-gpu-prefill-host (8 prefill tests, host model and simulator) and
  check-own30b-decode-profile with tools/own30b_decode_profile.c.

CUDA++ code that was ahead stays: the fp8 upload owner generation and the
test_out_open 0600 receipt in gpu_fp8gemm_test. The baseline records the
4376c21 blobs and names those paths. Not carried: the
require_physics_pin.sh line in omega's chip runners (no such runners here).

This branch first merged main, which already holds the b5ab3fe sync
(#207); the tree took main's side for every file of the earlier b5ab3fe
commits. Not kept from 063a0c2: the dagsim mutant FAIL-line tightening
and the omega_runtime_api.internal abort_token row (main declares it in
the manifest as EXPERIMENTAL).

Co-authored-by: Claude Opus 5.5 (via omp) <[email protected]>
@aien-spark
aien-spark merged commit 769997d into main Oct 31, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants