Skip to content

gpu-gate: the cancellation self-tests signal once the timeout sleeps past its fork - #1035

Merged
juliankang4 merged 3 commits into
midagedev:mainfrom
juliankang4:dgx-spark-q38-fix-gate-cancel-status
Oct 31, 2026
Merged

juliankang4 merged 3 commits into
midagedev:mainfrom
juliankang4:dgx-spark-q38-fix-gate-cancel-status

Conversation

@juliankang4

Copy link
Copy Markdown
Collaborator

gpu-gate.sh --self-test의 취소 테스트가 GNU timeout 9.4에서 가끔 exit 7 대신 143을 받던 경합을 고쳤습니다.

What and why

tools/gpu-gate.sh --self-test sometimes failed in CI because a cancelled run reported 143, where the test expects the binary's own code (7 for the held-run cases, 23 for the lib-path signal case). The cause is a race in GNU coreutils timeout before 9.5, which is what Ubuntu 24.04 ships (9.4). The cases signalled the outer timeout as soon as the stub printed up, but that line does not mean timeout has returned from its fork(). A signal that arrives before it returns is not forwarded: 9.4's cleanup() runs _exit (128 + sig) when it has no child pid yet, and 9.5 fixed that.

The two cases now send the signal only after timeout is asleep, which is after its fork has returned. They reuse cancel_ready from tools/ref/cancel-stubs.sh, which the held-run case already needed for the watch cases. The expected codes are unchanged, and 143 is never accepted. tools/gate-bound.sh gets one comment sentence that says GNU timeout before 9.5 reports 143 for a TERM that comes before its fork returns, and that the group TERM ends the child.

Release notes: internal: the gpu-gate self-test no longer fails on runners with GNU timeout before 9.5.

Code shape

  • No second copy. The wait reuses cancel_ready from tools/ref/cancel-stubs.sh; no new helper.
  • Model code holds only architecture facts. No model code is touched.
  • Dead code is gone.
  • No silent failure. The cases still fail on a wrong exit code; a wait that never sees timeout asleep ends at the existing failsafe bound and the case then fails.
  • Comments state what is true now.
  • No relaxed gate. No expected code, band or tolerance moved.
  • cargo fmt and cargo clippy are clean on what I touched; the lint count did not go up.

Duplication found or removed (path:line): none.

What I ran

The race, on a DGX Spark with GNU timeout 9.4 (Ubuntu 24.04), with three yes loops and a small script pinned to one core (taskset -c 0). The script starts timeout 100 <stub>, where the stub traps TERM with exit 7, waits for the stub's up line, sends timeout one TERM and counts the exit codes. The first two rows signal on up, which is what the cases did. The last row also waits until timeout is asleep (process state S), which is what the cases do now:

signal when the stub is up (300 starts):         {'exit 7': 265, 'exit 143': 35}
signal when the stub is up (400 starts):         {'exit 7': 397, 'exit 143': 3}
signal after timeout is asleep (400 starts):     {'exit 7': 400}

The changed cases, on the head, on an RTX 5090 machine (GNU timeout 9.7, so the race itself does not occur there), under CPU contention (taskset -c 0,1, GPU_GATE_SELFTEST_LOAD=6):

for k in 4 1; do GPU_GATE_SELFTEST_LOAD=6 taskset -c 0,1 bash tools/gpu-gate.sh --self-test --part $k; done
part 4 rc=0 78 s
ok lib path propagates signal
ok lib child is stopped after signal
gpu-gate self-test: 60 of 60 ok
part 1 rc=0 33 s
ok gate()'s bound ends a held run with its binary, output and lock, the binary's code kept
ok gate()'s bound ends a held run under the watch with its binary, output and lock, the binary's code kept
gpu-gate self-test: 69 of 69 ok

The held-run cases on a DGX Spark with GNU timeout 9.4, --self-test --part 1 four times each on the base and on the change, same contention: all eight runs rc=0, both held-run cases ok.

The change touches only self-test code and a comment, so no card gate applies.

AI help

AI 코딩 에이전트가 코드를 작성한 후 설명을 썼고, 테스트를 마쳤습니다. diff 리뷰와 PR 초안 리뷰는 제가 직접 했습니다.

@github-actions github-actions Bot added the tools Tools, recipes, CI and the toolchain: tools/, justfile, .github/, Cargo label Oct 31, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: CLEAN: No behavior or contract defect was found in the changed self-tests.

Checked

Duplicates: 0 new functions, types or constants; the waits reuse cancel_ready. No dryer pairs were available.
Complexity: no Rust functions changed; the added shell branches wait for monitor readiness.
Silent failure: 3 readiness probes followed through retries, safety bounds and the final exit-code and cleanup assertions.
Tests (static mutation check): dropping signal forwarding at tools/gate-bound.sh:195 breaks the held-run, watched-run and library exit-code assertions. Adding a cancel before the return at tools/boxq.py:575 fails the revised cancel-log assertion; its matcher was checked in memory against cancel, poll and tail commands.
Validation: all 3 changed scripts pass bash -n; the changed Python fixture parses.
Gates: the cancel-log matcher has a dated PIN and a false-positive reason. Expected exits 7 and 23 remain unchanged.
Escape catalog: no C1–C4 pattern matched the changed code.

Notes

Dryer and crapper did not run because no Rust file changed. Full self-tests were not run during this review.

🔨 Shingler, bloomery's review bot: approves a clean pull request and requests changes on a must-fix (how it works)

@juliankang4

Copy link
Copy Markdown
Collaborator Author

The review is done and the contribution check passed, so I'm merging this.

@juliankang4
juliankang4 merged commit 5658c58 into midagedev:main Oct 31, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tools Tools, recipes, CI and the toolchain: tools/, justfile, .github/, Cargo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant