Skip to content

test: make process-group timeout test deterministic - #32

Merged
dezgit2025 merged 1 commit into
mainfrom
test/deterministic-timeout-kill
Oct 1, 2026
Merged

dezgit2025 merged 1 commit into
mainfrom
test/deterministic-timeout-kill

Conversation

@dezgit2025

Copy link
Copy Markdown
Owner

Fixes the flaky required check V5 Stage C / macOS ARM64, caused by test_real_subprocess_timeout_kills_child_process_group.

Root cause

The test raced a child that writes a sentinel after sleep(1) against a 1 s timeout. On an idle Mac the child would have written 33–51 ms after the group kill returned (15 measurements), so scheduling jitter on CI runners flipped the result. The runner code (_bounded_subprocess → os.killpg(SIGKILL) on timeout) was correct.

Fix

  • The child sleeps 30 s, so it can't win the race; the parent writes the child's PID to a file right after Popen.
  • After TimeoutExpired, the test polls os.kill(pid, 0) until the PID is gone (5 s deadline), then asserts that no sentinel was written.
  • The timeout goes from 1 s to 2 s so a slow runner has time to start the parent. Test time is about 2.0 s, the same as before (1 s timeout + 1.2 s sleep).

Verification

Check Result
New test, idle, CI flags (PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 -W error) 25/25 pass
New test with all CPU cores busy 20/20 pass
Known-bad: killpg broken so only the parent dies fails with child survived the group kill
Full suite (src/ + verify/test_codex_release_upgrade.py), ruff 577 passed, clean
macOS CI job re-run until 10 consecutive passes (results in a comment)

Test-only change: no regression-gate latency run needed (CLAUDE.md rule 13).

The test raced a child that wrote a sentinel after 1 s against a 1 s
timeout. Measured margin on an idle Mac was 33-51 ms, so scheduling
jitter on CI runners flipped the result and the required macOS check
failed randomly.

The child now sleeps 30 s and the parent records its PID. After the
timeout the test polls until that PID is gone (5 s deadline) and still
asserts no sentinel was written. A broken group kill (killpg failing,
only the parent killed) fails the test with 'child survived the group
kill'.
@dezgit2025

Copy link
Copy Markdown
Owner Author

macOS CI: 10/10 consecutive passes of V5 Stage C / macOS ARM64 (run 36881836762, attempts 1–10, jobs 110435272404 … 110443425943). All 9 required checks green.

@dezgit2025
dezgit2025 merged commit 17b0ed0 into main Oct 1, 2026
93 checks passed
@dezgit2025
dezgit2025 deleted the test/deterministic-timeout-kill branch October 1, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant