[nirvedhmeshram]

Problem

The runner has 4 job slots for 8 GPUs, and a job waiting for GPUs still holds its slot. With one fixed GPU request per job, a common state is one 2-GPU job running and three 8-GPU jobs spinning in "Acquire GPUs". That leaves 6 GPUs idle, while the 1/2/4-GPU jobs that could use them wait in GitHub's queue, which we can't reorder.

Measured over 78 jobs from the 20 most recent completed GPU workflow runs:

GPUs	Jobs	Avg wait for GPUs	Avg test time
1	10	42 s	197 s
2	22	38 s	236 s
4	12	40 s	258 s
8	30	698 s	213 s

GPUs were 58% utilized while CI was busy, and 16.6 of 40.4 GPU-hours sat idle while some job was spinning on acquire. Full Iris Tests runs took 89 and 115 min, against roughly 60 and 79 min of GPU work if perfectly packed.

Change

gpu_task_queue.py: a job gets a list of <test_dir>:<ranks> tasks, runs them concurrently, and takes GPUs per task. It uses the same bitmap and flock as gpu_allocator.sh, so it coexists with the perf and external-validation jobs that still use the bash allocator.

• Tasks are started largest-first; smaller tasks fill whatever GPUs are free, and a task's GPUs go back the moment it ends.
• Anti-starvation: once the largest pending task has waited 5 min, it posts a reservation (/tmp/iris_gpu_state.reserve). New work then stops everywhere, including in the bash allocator, until the reserved task fits. A larger request takes over a smaller one's reservation; the owner refreshes it, and it expires after 10 min if the owner dies.
• Each task runs in a private copy of the checkout, because editable and pip installs write into the source tree. The copy includes .git, which setuptools-scm needs. It's about 21 MB.
• Per-task logs are printed in collapsible groups as each task finishes, with a pass/fail table in the step summary. The job fails if any task fails.
• On cancel or timeout, running tasks are killed, their partial logs printed, and their GPUs returned. If the process is killed outright, the existing Release GPUs always() step returns its GPUs using a held-mask file.

iris-tests.yml restructuring:

• git installs: one job per test dir runs 1/2/4/8 ranks (20 jobs become 5)
• editable and pip: one job each, covering all dirs at 2 ranks (10 jobs become 2)
• new examples: one job runs 2/4/8 ranks (3 jobs become 1)

gpu_allocator.sh:

• polls every 10 s instead of 60 s (still a 3 h limit)
• respects the reservation

gpu-reset.yml now also clears the reservation file.

Trade-offs

• You get one check per test dir or install type instead of one per rank count. Each task still has its own log group and summary row, and a failure is annotated with the task name.
• Re-running a failed job re-runs its whole bundle.
• More tasks can run at once than before (up to 8 one-rank tasks rather than 4 jobs), so more Apptainer overlays can exist at the same time.

Testing

Tested locally with stub tasks that sleep instead of using GPUs:

• packing with 8/4/2/1 tasks plus one that fails
• two queues contending, where an 8-GPU task takes over a 2-GPU task's reservation instead of being starved, and the bash allocator waits on the reservation
• SIGTERM mid-run (GPUs, temp dirs, and child processes cleaned up)
• SIGKILL followed by the release_gpus.sh fallback

Not yet verified on the real runners. This PR's own CI run is the first real test.

🤖 Generated with Claude Code

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 2 · 2 · 1

Open (5)

• exit 1 inside acquire_gpus() will terminate the entire script (and likely the workflow step)… · New
• The held-mask file path defaults to a fixed filename (.../iris_gpu_queue_held). If RUNNER_TEMP… · New
• Since HELD_FILE is used as crash-recovery state for releasing GPUs after an abrupt termination,… · New
• Copying the entire workspace (including .git) per task can become a noticeable IO/disk bottleneck… · New
• Creating a directory with mkdtemp() and immediately deleting it to satisfy copytree() is… · New

What changed in this PR

This PR restructures the GPU CI workflow to reduce runner-slot blockage by allocating GPUs per task (within a job) rather than per job, with an anti-starvation reservation mechanism shared between the new Python queue and the existing bash allocator.

Changes:

• Replace matrix-per-rank workflow jobs with bundled per-directory (and per-install-type) jobs that run multiple rank tasks concurrently via gpu_task_queue.py.
• Introduce a Python task queue/allocator that shares bitmap + flock protocol with gpu_allocator.sh, plus reservation support to prevent starvation of large (e.g., 8-GPU) tasks.
• Update the bash allocator to poll more frequently and to respect node reservations; update reset logic to clear reservations; enhance GPU release to recover allocations after abrupt queue termination.

File	Description
.github/​workflows/​iris-tests.yml	Bundles multiple rank runs per directory/install type and invokes gpu_task_queue.py instead of per-job GPU acquisition.
.github/​workflows/​gpu-reset.yml	Resets allocator reservation state in addition to the bitmap.
.github/​scripts/​release_gpus.sh	Adds fallback release path using a held-mask file written by gpu_task_queue.py.
.github/​scripts/​gpu_task_queue.py	New per-task GPU allocator/runner with reservation-based anti-starvation and per-task workspaces/logging.
.github/​scripts/​gpu_allocator.sh	Reduces polling interval and blocks new allocations while a reservation is active.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[mawad-amd]

@nirvedhmeshram noticed not all the GPU workflows are using the new queue. Only iris-tests.yml got converted. Perf Regression and External Validation are still on the per-job bash allocator, which is ~35% of GPU job-minutes since Oct 6.

Also, can we do an eval of current vs before?

[nirvedhmeshram]

@mawad-amd thanks. Both points are addressed in #578, which moves perf, external validation, nightly and the CCL benchmark onto the queue.

Before/after #571. The after-window is only ~1.5 days (9 Iris Tests runs, 49 perf jobs), so read these as indicative.

• Iris Tests: runner-slot minutes per run went from 367 to 191 (median), and the share of slot time spent waiting for GPUs from 57% to 38%. End-to-end median is unchanged (433 vs 430 min); p90 improved from 679 to 488 min. GPU utilization while the host is busy went from 71% to 86%.
• Jobs still on the bash allocator got worse. They honour the queue's node reservations but never post their own, so their 8-GPU requests lose out. Perf GPU wait went from 7 to 25 min (median) and from 50 to 132 min (p90). 41% of perf wait time after the merge was spent blocked by a reservation. External validation p90 went from 4 to 51 min. The increase holds on comparably busy days, so it isn't just load.
• Your ~35%: perf + external validation were ~31% of runner-slot time since Oct 6, but held only ~6% of GPU-hours. Nearly all of that slot time was spent waiting for GPUs.

#578 also fixes the Triton external validation test, which never passed its GPUs to the container and could see all 8. I'll re-run this comparison once there's a week of data with #578 in.

🤖 Generated with Claude Code