Skip to content

CI: run the unit-test files in parallel - #224

Open
davidwalter2 wants to merge 2 commits into
WMass:mainfrom
davidwalter2:261105_parallelUnitTests
Open

davidwalter2 wants to merge 2 commits into
WMass:mainfrom
davidwalter2:261105_parallelUnitTests

Conversation

@davidwalter2

Copy link
Copy Markdown
Collaborator

all-unit-tests runs the 41 collected test files one after the other, and every fit job (fitting, sparse-fits, alternative-fits, regularization, bsm, likelihoodscans, abcd-fits) waits for it. In #208's last CI run it took 56 of the 63.5 min; everything after it finished within 7 min.

The files now run 12 at a time:

  • Still one process per file. XLA keeps compiled code mapped until the process exits, and the suite in one process exceeds vm.max_map_count. That is also why this uses xargs -P and not pytest-xdist, whose workers live across files.
  • Each process is capped at 4 threads (OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, TF_NUM_INTRAOP_THREADS; 2 inter-op threads). Uncapped, every TensorFlow process would start one thread per core of the 256-core runner host, which the WRemnants CI shares. 12 × 4 threads is 48 cores. rabbit sets no thread counts itself, so the caps also reach the rabbit_fit.py subprocesses some tests start.
  • Logs. Each file's output goes to its own log and is printed in one group after all files have run, as before. A line per file as it ends shows progress meanwhile.
  • Timeout. A file still running after 30 min is killed and counts as failed (exit 124), instead of holding the job for the default 6 h.
  • Unchanged: collection over tests/ decides what runs (test_ci_registration passes on the new workflow), every file runs even if another fails, and the job fails if any did.

Running files in parallel is safe: the tests write into their own temporary directories, and the one that uses $RABBIT_OUTDIR (test_global_asym_impacts.py) writes to its own subdirectory.

Checked on a 64-core submit node, with the step's script and env taken from this workflow and run under bash -e:

  • the whole suite passes in 412 s;
  • a toy suite with a passing, a failing and a timed-out file reports exit 0, 1 and 124, lists the two failures, and fails the step.

Expected on the runner host: per file it ran ~1.7× slower than the submit node, which puts all-unit-tests at ~8–10 min and the whole CI at ~15–17 min instead of ~63. The floor is the slowest file, test_mcstat_kfold.py (7.6 min in CI). This PR's own CI run shows the real timing.

Not changed: the fit jobs still wait for all-unit-tests. They only need make-tensor's files, so dropping that wait would overlap them with the unit tests (CI ~10 min), at the cost of running fits for PRs whose unit tests fail.

🤖 Generated with Claude Code

https://claude.ai/code/session_014i9SdYoFtmw6ig5a4kzAx2

davidwalter2 and others added 2 commits November 5, 2026 09:26
all-unit-tests ran the 41 collected test files one after the other, in
~55 min, and every fit job waits for it: 56 of the 63.5 min of WMass#208's
last CI run. The files now run 12 at a time, still one process each for
the XLA mappings, with each process capped at 4 threads: uncapped,
every TensorFlow process would start one thread per core of the
256-core runner host. Each file's output goes to its own log, printed in
one group after all files have run, with a line per file as it ends. A
file still running after 30 min is killed and fails the job.

Checked on a 64-core submit node with the step's script and env taken
from this workflow and run under bash -e: the whole suite passes in
412 s. On a toy suite it reports a failing and a timed-out file
(exit 1 and 124) and fails. test_ci_registration passes on the new
workflow.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_014i9SdYoFtmw6ig5a4kzAx2

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Issues Found

No issues found.

Summary

Total issues: 0 critical, 0 important, 0 minor.

Overall Verdict

Status: Patch is correct

Explanation: The xargs-based parallelization is mechanically sound: filenames are passed safely as single argv entries, per-file exit codes are captured in .rc files and correctly aggregated (including the missing-file and timeout-exit-124 cases), and the collect-only line still satisfies the repo's workflow-guard tests in test_ci_registration.py (whole-directory coverage token present, no narrowing flags). Log-name flattening is injective because tests/ is flat, the glob-empty edge case still ends in job failure via failed="collection", and all fallible commands are guarded against GitHub's bash -e shell. Thread caps, per-file timeout, and deferred grouped output are documented, intentional design choices.

Review details
  • Model: cern/Kimi-K3-FLATIRON

Review metrics: 4 turns · 6 tool calls · 14m 48s

  • Tokens: 71.7K input · 41.2K cached (58%) · 23.6K output

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant