[drakestapleton]

Purpose and scope

Pull requests now run the fresh-checkout host commands from docs/START_HERE.md, with a written rule for what each result means. Refs #90.

New files:

• .github/workflows/host-gates.yml
• tools/ci/host-gates.sh
• docs/CI_ACCEPTANCE.md

Small pointers in docs/START_HERE.md, CONTRIBUTING.md, and AGENTS.md.

This does not edit ci.yml, runtime-api.yml, the Makefile, or any file from PR #170. It does not touch the frozen C stack, src/omega_cubin.*, or om/cubin/.

How this composes with #189 and the x86 job

• make test stays the host-tests job in .github/workflows/ci.yml. This pull request does not run that suite a second time. The arm job still installs pinned nvdisasm 13.0.85 there, and still judges the exit code and the final CUDA++ HOST TESTS PASS line. Intentional FAIL and ERROR lines in that log stay visible.
• make test-runtime-api stays .github/workflows/runtime-api.yml. Issue CI: run make test-runtime-api on PRs and main (ADR 0043 follow-up) #189 owns that job. This pull request does not add another copy.
• x86-64 make test is the x86-host-tests job in .github/workflows/host-x86.yml (PR CI: add an x86-64 host-tests job (portable suite + freestanding link) #199). This pull request does not add that job and does not change the AArch64 host-tests job. On x86-64 the portable suite is the contract in docs/OMEGA_HOST_TESTS.md. The device open returns ENOTSUP and requires AArch64 Linux. That is not a supported GPU run.

What each gate runs

Gate	Job	Pass condition
make -j4	host-gates / fresh-checkout on ubuntu-24.04-arm, 30 minute limit	Exit 0. Both archives exist. sha256 uploaded.
make test	existing ci.yml / host-tests	Exit 0 and the final PASS line.
make test-native-artifact	fresh-checkout	Exit 0 and test_gpu_artifact_export: <n> checks, 0 failed.
make zero-cuda-gate	fresh-checkout	Exit 0. Log keeps ZERO_CUDA FAIL for the stub libcudart probe, then ZERO_CUDA SELF-TEST PASS, then ZERO_CUDA PASS for the two binaries.
build/hello_gpu --sim	fresh-checkout	Exit 0, the simulator sentence, the 1048576-element check, the pinned output sha256, and the same hashes on a second run. If make -j4 exits 0 and build/hello_gpu is missing, the three simulator rows are FAIL.
build/hello_gpu --sim --bug=past-end	fresh-checkout	Exit 1, FAULT, 0 bytes past the end of buffer a, and no PASS line.
build/hello_gpu --bug=past-end	fresh-checkout	Exit 2. Refuses to leave the simulator.
make doctor	fresh-checkout	Exit 2 only. Every line matching `^[[:space:]]*FAIL([[:space:]]
python3 tools/figures/make_figures.py --check	host-gates / figures on ubuntu-latest, 5 minute limit	Exit 0 and figures current: <n>.
tools/ci/host-gates.sh --self-test	both jobs	Checker negative controls. The doctor cases reject an extra FAIL line, a summary count of 6, a missing allowlisted line, exit 1, exit 3, a FAIL line with no indent, and a tab before FAIL.

docs/img/stack.svg already matched the generator on main 522ee0d (figures current: 10). No SVG was rewritten.

CI on this pull request

GitHub tested head d558e5d06eb5f9c847dc8bd28c50c76d862c2acb.

host-gates passed. fresh-checkout last line: CUDA++ HOST GATES PASS. The self-test printed PASS doctor-fail-noindent, PASS doctor-fail-tab, and CUDA++ HOST GATES SELF-TEST PASS. make doctor exited 2 and was recorded PASS. Chip qualification stays NOT_RUN.

CI host-tests passed in 15m3s. Last line: CUDA++ HOST TESTS PASS (62 targets).

x86-host-tests passed in 7m47s.

The other checks also passed: runtime-api, hello-sim, library, drift, om-units, sbom, audit-scanner, and CodeQL.

Known limitations

• make hello-gpu, chip tests, and any open of /dev/nvidia* or /dev/aien-gpu are NOT_RUN. The runners have no GB10.
• The Inkscape legend test is NOT_RUN. Inkscape is not installed on the figure job.
• An outside engineer following docs/START_HERE.md on a machine this repository does not control is NOT_RUN.
• The ruleset main-verified-promotion still has no required status checks. Marking a check required is an administrator change and was not done.
• This workflow does not run x86-64 make test and does not set CUDAPP_MEM_RESERVE_MIB. That suite is the x86-host-tests job in .github/workflows/host-x86.yml.

— Grokbot

  

[drakestapleton]

Review at cbcbcab: needs changes (small). CI hasn't started yet: all 13 checks were queued at review time, so fresh-checkout has no AArch64 result yet.

Provenance. This is 1 commit from cloud agent bc-a927ce42, branched off main 522ee0d. The body says Refs #90, which matches the issue. It doesn't close #90, and that's correct, since the ruleset's required checks are left to an admin.

Scope. Existing workflows aren't touched (ci.yml and runtime-api.yml are unchanged). It adds a new file, .github/workflows/host-gates.yml, with two jobs:

• figures (L21-37): runs on ubuntu-latest with a 5-minute limit.
• fresh-checkout (L38-56): runs on ubuntu-24.04-arm with a 30-minute limit.

Triggers are pull_request plus pushes to main, and permissions are contents: read. checkout and upload-artifact are pinned to SHAs. concurrency cancels in-progress runs per ref.

Also changed: the new tools/ci/host-gates.sh (415 lines) and docs/CI_ACCEPTANCE.md, plus pointer lines in AGENTS.md L78, CONTRIBUTING.md L24-25 and docs/START_HERE.md L93-97. The check names figures and fresh-checkout are new, and none collides with host-tests, runtime-api or the others.

Fake-green check. This is mostly strict.

• run_log captures rc. finish exits 1 on any FAIL (L250-259).
• Each checker requires the exact exit code and its marker lines: artifact 0 failed, ZERO_CUDA self-test plus both binaries, sim pinned output sha with two identical runs, past-end exit 1 with no PASS, GPU bug path exit 2, doctor exit 2 with NOT READY and no READY. line, and figures with no stale.
• --self-test covers the reject cases.
• The || true at L46, L365 and L366 only guards greps whose result is then checked.
• The if: always() lines are on the artifact upload only.

One gap fails open:

• tools/ci/host-gates.sh L356 and L391-393. If make -j4 passes but build/hello_gpu isn't built, the three sim and bug gates are recorded NOT_RUN. record doesn't set fail for NOT_RUN, so the job ends with CUDA++ HOST GATES PASS with no sim run at all. The same pattern sits at L352-353, but there it's harmless, because the make -j4 failure already sets fail. Please make a missing build/hello_gpu after a successful make -j4 a FAIL, and add a self-test case for it.
• Minor: doctor_ok matches make's Error 1 text (L295). That's fine, just brittle across make versions.

GB10. There's no claim beyond NOT_RUN. make hello-gpu, chip tests and device opens are NOT_RUN in both the script (L321) and the doc. The local evidence is x86-64 portable only, so the arm runner result is still pending.

Fit with #197 and #196. There are no shared files: #197 only edits runtime-api.yml. #196 is still an issue, with no PR open. Its x86 job should take a name other than host-tests, fresh-checkout, figures and runtime-api. If it edits ci.yml, it won't conflict with this PR. Suggested merge order: #197, then #198, then #196. No textual rebase is expected. #196's PR should update the pointers here (host-gates.yml L15, the script's x86-64 make test NOT_RUN note at L320, and docs/CI_ACCEPTANCE.md L9).

Follow-ups. The body lists none to file. It notes these as NOT_RUN or admin-only: the Inkscape legend test, an outside-engineer run, required status checks on main-verified-promotion, and x86 make test (#196).

— Grokbot

[drakestapleton]

Re-review at b7bc8da: needs changes (doctor gate only).

The branch has 2 commits: cbcbcab, then b7bc8da on top of it. It wasn't force-pushed. The base is now 86b78f2, after #199 merged. CI is green on all 14 checks: host-gates run 41080804127 (figures, fresh-checkout), ci run 41080804265, and the rest.

Fixed

• hello_gpu missing. tools/ci/host-gates.sh L117-130 and L325-341: a successful make -j4 with no build/hello_gpu now records 3 FAILs and sets fail=1. NOT_RUN is kept only when make itself failed.
• Self-tests. L256-287 feed real cases: a nonexistent path, an empty executable file, and a make rc of 2. They assert both fail and the FAIL/NOT_RUN row counts, and saved_fail carries any failure into the exit 1 at L295-299. CI runs --self-test in both jobs (host-gates.yml), and a self-test failure fails the job. The || true at L257, L277 and L278-279 only stops set -e from aborting before the asserts run. The return code and counts are still checked.
• Error 1 text. It's gone from doctor_ok (L100-115), and L246-247 tests that the text alone doesn't pass.
• x86 pointer. L395 now says ENFORCED_ELSEWHERE with x86-host-tests in host-x86.yml.

Doctor gate: doesn't meet the policy yet (L100-115). Exit codes are handled right: only rc 2 passes, and 0, 1, 3 and anything else fail at L102. The output check is loose, though:

• It only checks that the five expected FAIL lines are present, using grep -F substring matches (L104-108). There's no check that they're the only ones, so an unexpected sixth FAIL ... line still passes. That's a fixed list of required lines, not an allowlist that rejects anything else.
• The count isn't pinned. L109 accepts any NOT READY:, so NOT READY: 9 problems above passes.
• The self-tests have no case for an unexpected problem line, a wrong count, or rc 1 or rc 3. Only rc 0 and the READY line are covered (L244-249).

What's needed:

1. Take every line matching ^ FAIL and require it to equal one of the five allowlisted lines exactly. Those are the kernel module line (full text, including the /proc/... parenthesis), nvidiactl, nvidia0, nvidia-uvm, and no NVIDIA GPU. Any other FAIL line fails the gate.
2. Require exactly 5 FAIL lines and an exact NOT READY: 5 problems above. prefix.
3. Add self-test rejects for an extra unexpected FAIL line, a count of 6, a missing allowlisted line, rc 1 and rc 3.

Fake-green re-scan. It's clean: no continue-on-error, and no if: skip except always() on the artifact uploads. Every || true sits on a grep or count that's checked afterwards. GB10 stays NOT_RUN (L396), and the PR makes no chip claim.

#199 is merged and #197 is still open. The pointers are correct now.

— Grokbot

[drakestapleton]

Re-review at 16ff7c1: needs changes, one small gap left in how FAIL lines are found.

This head is a fast-forward from b7bc8da: it is 1 commit ahead and 0 behind, so nothing was force-pushed. The only file changed is tools/ci/host-gates.sh (+57/-8). The PR is mergeable and clean on base bac7330. All 15 checks are green at 16ff7c1: host-gates run 41084623303 (figures, fresh-checkout), x86-host-tests 41084623395, ci 41084623539, runtime-api 41084623271, hello-sim 41084623681, library 41084623293, drift 41084623479, and CodeQL 41084617093.

I ran the doctor_ok function from L103-153 against edge cases.

Works

• Exit code (L124). Only exit 2 is accepted. Exit 0, 1 and 3 are all rejected.
• Allowlist (L103-119). A case statement compares the whole line against the five allowlisted lines exactly.
  • Duplicates and missing lines. The total must be exactly 5 (L146) and each kind must appear once (L147), so a repeated line can't stand in for a missing one. I tested this: replacing uvm with a second nvidia0 is rejected.
  • Whitespace and partial matches. A trailing space, a longer line, or FAIL x with three spaces is rejected.
• Summary line (L148-151). The pattern ^NOT READY: 5 problems above\. must match exactly once, and there must be exactly one ^NOT READY: line.
  • Rejected: 6 problems, 50 problems, bare NOT READY: 5, an indented summary, a duplicated summary, and a missing summary.
  • 50 can't match, because the literal problems must come straight after the 5.
• Self-tests. These are real. expect_reject (L190-198) sets fail=1 whenever the checker accepts a case it should reject, and that leads to the exit 1 at L347-348.
  • doctor-extra-fail (L291-293) adds a sixth FAIL line.
  • doctor-count-6 (L294-295) uses sed to change the summary to 6.
  • doctor-missing-allowlisted-line (L296-297) drops the uvm line with grep -vxF.
  • doctor-exit1 and doctor-exit3 (L298-299) pass exit 1 and exit 3.

Gap (L132). FAIL lines are found with ' FAIL '*, which only catches lines starting with exactly two spaces, FAIL and two spaces. A FAIL line in any other form is never counted, so the gate still passes. I tested each of these appended to the good fixture, and each was ACCEPTED:

• FAIL something (no leading spaces)
• FAIL something (one leading space)
• a tab followed by FAIL x

Fix. Treat any line matching ^[[:space:]]*FAIL([[:space:]]|:|$) as a problem line, so it must pass doctor_fail_line or the gate fails. Then add a self-test reject for one of these variant forms.

Fake-green scan. It's still clean. There's no continue-on-error, and no if: skip apart from always() on the uploads. The new || true on the grep -c calls (L148-149) feeds counts that are checked right after. GB10 stays NOT_RUN.

— Grokbot

[drakestapleton]

Re-review at d558e5d: ready to merge.

Diff. This head is a fast-forward from 16ff7c1: 1 commit ahead, 0 behind, so nothing was force-pushed. The only file changed is tools/ci/host-gates.sh (+10/-5):

• the comment at L98-99;
• the FAIL-line check at L132, which is now [[ "$line" =~ ^[[:space:]]*FAIL([[:space:]]|:|$) ]] instead of the case prefix;
• two new self-tests at L297-304.

Nothing else changed.

Edge cases. I ran the new doctor_ok (L104-155) on these:

• The good five-line output with exit 2 PASSES.
• Each of these is REJECTED:
  • FAIL x with no indent;
  • FAIL x with one space;
  • a tab followed by FAIL x;
  • FAIL: x;
  • a bare FAIL, indented or not;
  • an allowlisted line with a trailing space;
  • FAIL x with three spaces;
  • a duplicated line standing in for uvm;
  • a missing uvm line;
  • a summary count of 6 or 50;
  • a missing summary;
  • exit 0, 1 or 3.
• Lines starting FAILED, FAILURE or a lowercase fail aren't treated as problem lines, as the regex intends, so the gate still passes. That's acceptable: the doctor prints problems as FAIL . Note it for later if the doctor's output format changes.

Self-tests. Both are real:

• doctor-fail-noindent (L299-301) appends FAIL x.
• doctor-fail-tab (L302-304) appends a tab followed by FAIL x.

If either is accepted, expect_reject (L189-197) sets fail=1, which ends in SELF-TEST FAIL and exit 1 (L352-353). --self-test runs in both host-gates jobs.

CI. All 15 checks at d558e5d are green: host-gates 41087598473 (figures, fresh-checkout), x86-host-tests 41087598346, ci 41087598565, runtime-api 41087598377, hello-sim 41087598416, library 41087598383, drift 41087598347, and CodeQL 41087589254. The PR is mergeable clean on base bac7330, and #197 merged at 10:53 AM CT.

Fake-green scan. Still clean: no continue-on-error, and no if: skip apart from always() on the uploads. Every || true is followed by a check. GB10 stays NOT_RUN.

— Grokbot