Repository navigation
CI: run the unit-test files in parallel - #224
davidwalter2 wants to merge 2 commits into
Conversation
all-unit-tests ran the 41 collected test files one after the other, in ~55 min, and every fit job waits for it: 56 of the 63.5 min of WMass#208's last CI run. The files now run 12 at a time, still one process each for the XLA mappings, with each process capped at 4 threads: uncapped, every TensorFlow process would start one thread per core of the 256-core runner host. Each file's output goes to its own log, printed in one group after all files have run, with a line per file as it ends. A file still running after 30 min is killed and fails the job. Checked on a 64-core submit node with the step's script and env taken from this workflow and run under bash -e: the whole suite passes in 412 s. On a toy suite it reports a failing and a timed-out file (exit 1 and 124) and fails. test_ci_registration passes on the new workflow. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_014i9SdYoFtmw6ig5a4kzAx2
There was a problem hiding this comment.
Issues Found
No issues found.
Summary
Total issues: 0 critical, 0 important, 0 minor.
Overall Verdict
Status: Patch is correct
Explanation: The xargs-based parallelization is mechanically sound: filenames are passed safely as single argv entries, per-file exit codes are captured in .rc files and correctly aggregated (including the missing-file and timeout-exit-124 cases), and the collect-only line still satisfies the repo's workflow-guard tests in test_ci_registration.py (whole-directory coverage token present, no narrowing flags). Log-name flattening is injective because tests/ is flat, the glob-empty edge case still ends in job failure via failed="collection", and all fallible commands are guarded against GitHub's bash -e shell. Thread caps, per-file timeout, and deferred grouped output are documented, intentional design choices.
Review details
- Model:
cern/Kimi-K3-FLATIRON
Review metrics: 4 turns · 6 tool calls · 14m 48s
- Tokens:
71.7Kinput ·41.2Kcached (58%) ·23.6Koutput
all-unit-testsruns the 41 collected test files one after the other, and every fit job (fitting,sparse-fits,alternative-fits,regularization,bsm,likelihoodscans,abcd-fits) waits for it. In #208's last CI run it took 56 of the 63.5 min; everything after it finished within 7 min.The files now run 12 at a time:
vm.max_map_count. That is also why this usesxargs -Pand not pytest-xdist, whose workers live across files.OMP_NUM_THREADS,OPENBLAS_NUM_THREADS,MKL_NUM_THREADS,TF_NUM_INTRAOP_THREADS; 2 inter-op threads). Uncapped, every TensorFlow process would start one thread per core of the 256-core runner host, which the WRemnants CI shares. 12 × 4 threads is 48 cores. rabbit sets no thread counts itself, so the caps also reach therabbit_fit.pysubprocesses some tests start.tests/decides what runs (test_ci_registrationpasses on the new workflow), every file runs even if another fails, and the job fails if any did.Running files in parallel is safe: the tests write into their own temporary directories, and the one that uses
$RABBIT_OUTDIR(test_global_asym_impacts.py) writes to its own subdirectory.Checked on a 64-core submit node, with the step's script and env taken from this workflow and run under
bash -e:Expected on the runner host: per file it ran ~1.7× slower than the submit node, which puts
all-unit-testsat ~8–10 min and the whole CI at ~15–17 min instead of ~63. The floor is the slowest file,test_mcstat_kfold.py(7.6 min in CI). This PR's own CI run shows the real timing.Not changed: the fit jobs still wait for
all-unit-tests. They only needmake-tensor's files, so dropping that wait would overlap them with the unit tests (CI ~10 min), at the cost of running fits for PRs whose unit tests fail.🤖 Generated with Claude Code
https://claude.ai/code/session_014i9SdYoFtmw6ig5a4kzAx2