Skip to content

Replace blind sleeps with explicit concurrency and shutdown-readiness checks - #128

Merged
valthon merged 2 commits into
masterfrom
fix/phase-group-parallel-test
Oct 21, 2026
Merged

valthon merged 2 commits into
masterfrom
fix/phase-group-parallel-test

Conversation

@valthon

@valthon valthon commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

What

Stop using blind sleeps as a proxy for verifying parallel activity and shutdown readiness. Check that tasks actually overlap and that the child process has installed its signal handler before exercising forced termination. This makes the checks reflect the behavior they are intended to verify.

The intent is to verify the existing guarantees more directly:

  • Parallel execution: prove that multiple tasks have started before any can finish, instead of assuming a shorter sleep will finish first. A serial runner must still fail these tests.
  • Concurrency limits: continue checking the configured peak number of tasks in flight, and add coverage for a parallel group with no concurrency limit.
  • Forced shutdown: wait until the test child confirms it is ready to ignore SIGINT, then verify forced termination and process closure, instead of guessing readiness or exit from elapsed sleep time.

Here, making the tests “deterministic” means removing timing races in the tests' observations. It does not impose a fixed execution or completion order on parallel work, serialize the framework, or replace real flow activity with mocks. Existing sequential-order, failure-handling, and timeout tests remain in place.

One motivating failure occurred in #127's CI (run 38503698391).

Parallel phase and group tests

ClusterBuildPhaseGroup › runs children concurrently when parallel ran a 40 ms step and a 5 ms step and asserted the 5 ms one finished first. That only holds if the two children start less than 35 ms apart. The runner does start them together (Bluebird.map with concurrency: Infinity), so the defect was the test's margin, not the runner. The phase's parallelize test made the same assertion, and the four cap and unbounded peak-in-flight tests relied on every body starting within sleep(10).

Each of these now records its start and waits on a rendezvous(parties) barrier that releases only once every party has started. The wait rejects after a named timeout (RendezvousTimeoutMs), so a non-concurrent runner fails the test instead of hanging it or passing. Peak counts are reached through the barrier, sized by a named cap. A group twin of the unbounded-concurrency test is added, because no group test checked that the runner honours an unbounded group.

rendezvous, planGatedStep and planInFlightTrackingStep (actor-first plan* factories with named run* runners and typed probe state in ctx.outputs) live in the orchestration fixture, with their own tests. Those cover release, hold until the last party arrives, rejection at the timeout, the default ceiling, and no timer left pending.

terminatePidsSync straggler test

The test slept 500 ms so bash could install trap '' INT before the SIGINT. On a loaded host the signal can arrive first, and then the SIGKILL path goes untested. The child now prints ready after installing the trap, and the test waits for that line. It then blocks in bash's built-in read rather than sleep 300. A sleep child either briefly empties /proc/<pid>/cmdline when exec'd, which the liveness check reads as "exited", or outlives the SIGKILL and holds the stdout pipe open, which keeps Jest from exiting. The test also fails fast if bash exits before printing ready. Readiness and closure each wait on the event with a 20 s ceiling, and a finally kills any surviving child, closes its stdin and awaits close, so a termination regression cannot leave the child holding Jest open. The fixed 200 ms post-kill sleep is gone.

Verification

  • Mutations of the runner:

    • forced serial: all six concurrency tests fail at the rendezvous timeout;
    • cap ignored, or off by one in either direction: both cap tests fail;
    • group hard-capped at 2: the new group unbounded test fails;
    • trap removed from the straggler: its test fails.
  • Repeat runs, before the last round of review fixes:

    • the orchestration suites passed 50/50 idle;
    • ProcessManager passed 30/30 idle and 20/20 with 32 CPU-burner processes;
    • in the reviewer's run, both passed 20/20 under 30 CPU-hog processes, with no leftover child processes and no open handles.

    The last round changed only test error paths: rejecting on a spawn error and on an early exit, and a fake-timer default-ceiling test. Since then each has passed once, in the per-commit gate below.

  • Per-commit gate: CI's test:ci sequence at each commit, without the ql-tool-app Jest project, whose tests never finish on the dev host (a jsdom synchronous XHR to localhost:80, which the host's WSL2 mirrored networking never refuses). CI runs them. The pre-commit hook was skipped (--no-verify) for the same reason.

    Commit RC Subject
    a59190d8 0 Wait for the straggler's readiness line instead of sleeping 500 ms
    394f46b2 0 Prove parallel groups run concurrently with a barrier, not sleep margins

    Current series, rebased on master ff3b06e1: at each commit a clean platform-root checkout ran install (sibling links verified), build, lint and every test:unit, all RC 0; at the tip the full cluster-tool Jest project passed 2,100/2,100. History check clean.

  • Review: an adversarial Codex review found two issues: the straggler child had no unconditional cleanup or readiness deadline, and the fixture factories used create* names and inline runners against STYLE.md. Both were fixed, folded into the commits above, and confirmed by a scoped re-review with no further findings. The mutation and repeat-run results above predate that fix.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PPcjf9SpvBzUMjiuGoKZt6
https://claude.ai/code/session_019LgGtRheR6Hac9WHX739y3

Rebased onto master 56a538b2 as 58dfc776, patch-identical (git range-diff). Gate on the rebased head: both commits install and build with lint; the tip ran pnpm run test:ci (4,489 Jest tests passed, every test:unit lane passed) with #132's fix applied only for the test step, RC 0.

@valthon
valthon requested a review from jglanz October 14, 2026 13:34
@valthon
valthon force-pushed the fix/phase-group-parallel-test branch from 7adf767 to 394f46b Compare October 15, 2026 00:42
@valthon valthon changed the title Make the parallel phase/group and straggler tests deterministic Replace blind sleeps with explicit concurrency and shutdown-readiness checks Oct 15, 2026
valthon and others added 2 commits October 20, 2026 14:16
The terminatePidsSync straggler test slept 500 ms to let bash install
`trap '' INT`. On a loaded host the SIGINT could land before the trap, so
the child exited gracefully and the SIGKILL path went untested.

The child now prints `ready` after installing the trap and the test
awaits that line. It blocks in `read` rather than `sleep 300`: an exec'd
or forked sleep either leaves /proc/<pid>/cmdline empty for a moment
(read as "exited" by the liveness check) or outlives the SIGKILL holding
the stdout pipe open, which kept jest from exiting.

Claude-Session: https://claude.ai/code/session_01PPcjf9SpvBzUMjiuGoKZt6
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019LgGtRheR6Hac9WHX739y3
"runs children concurrently when parallel" asserted ["fast","slow"] from a
5 ms versus 40 ms sleep, which only holds if the two children start less
than 35 ms apart. On a loaded runner they can start further apart and the
order flips (CI run 38503698391). The runner itself does start children
together; the test's wall-clock margin was the flake.

Each gated step now records its start, waits on a shared rendezvous that
releases only once every party has started, then records its end. A
runner that is not concurrent never releases the barrier, so the bounded
wait rejects and the test fails rather than hanging or passing.

The same sleep-ordering shape is fixed in the phase's parallelize test,
and the in-flight peak assertions (cap and unbounded, phase and group)
now reach their peak through the barrier instead of outlasting sleep(10).
A group twin of the unbounded-concurrency case is added.

rendezvous, createGatedStep and createInFlightTrackingStep live in the
orchestration fixture with their own unit tests.

Claude-Session: https://claude.ai/code/session_01PPcjf9SpvBzUMjiuGoKZt6
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019LgGtRheR6Hac9WHX739y3
@valthon
valthon force-pushed the fix/phase-group-parallel-test branch from 394f46b to 58dfc77 Compare October 20, 2026 19:00
@valthon
valthon merged commit 6c36dd0 into master Oct 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant