[nirvedhmeshram]

Follow-up to #571, addressing this comment. #571 moved only iris-tests.yml onto gpu_task_queue.py. Perf regression, external validation, nightly Triton and the CCL benchmark still used the per-job bash allocator.

Why it matters

A job on the bash allocator holds one of the runner's 4 slots while it waits for GPUs. The bash allocator also never posts a reservation, so its 8-GPU requests lose out to the queue's backfill of smaller tasks. From the last 6 successful runs of each workflow:

Workflow	Runs for	Waited for GPUs, holding a slot
Perf regression (4 jobs × 8 GPUs)	~75 s each	0 s – 8,557 s (2.4 h), 8 of 24 jobs over 50 min
External validation (2 jobs × 2 GPUs)	~60 s each	0 s – 3,093 s

Before/after #571

The runner is a single 8-GPU host with 4 job slots (iris-mi350x-0 to -3). At most 4 jobs ever ran at once across the four names. Comparison of every GPU workflow run since Sep 20; after #571 there are only ~1.5 days of data (9 Iris Tests runs, 49 perf jobs), so read these as indicative.

	Before #571	After #571
Iris Tests: runner-slot minutes per run, median	367	191
Iris Tests: share of slot time spent waiting for GPUs	57%	38%
Iris Tests: end-to-end, median / p90	433 / 679 min	430 / 488 min
GPU utilization while the host is busy	71%	86%
Perf: GPU wait, median / p90	7 / 50 min	25 / 132 min
External validation: GPU wait, median / p90	0 / 4 min	2.5 / 51 min

So #571 halved the slot time Iris Tests uses and raised GPU utilization. It also made the jobs left on the bash allocator wait longer:

• Reservations block them: they honour the queue's node reservations but never post their own. 41% of perf wait time after the merge was spent blocked by a reservation.
• Not just load: the increase holds on comparably busy days.
• Mostly waiting: since Oct 6, perf and external validation took ~31% of runner-slot time but held only ~6% of GPU-hours.

Change

gpu_task_queue.py gains --task RANKS NAME COMMAND, for GPU work that isn't a test directory. Each task still gets a private copy of the checkout, with GPU_DEVICES set to its GPUs. The existing TEST_DIR:RANKS form is unchanged.

Workflow	Before	After
Perf regression	4 jobs, 8 GPUs each	1 job, the 4 benchmarks back to back (~5 min), with the queue's node reservation
External validation	2 jobs, 2 GPUs each	1 job, Triton and Gluon side by side on 2 GPUs each
Nightly Triton	20 jobs, one per (dir, ranks)	5 jobs, one per dir with 8/4/2/1-rank tasks, same as iris-tests.yml
CCL benchmark	bash allocator	one queue task

Details:

• External validation bug: the Triton test acquired 2 GPUs but never passed --gpus to the container, so it saw all 8 and could land on GPUs another job held. The new run_external_validation.sh always passes them.
• Nightly: run_tests.sh takes a CONTAINER_IMAGE override, so the nightly Triton image goes through the same runner as everything else. The Triton version line moved into run_tests.sh.
• CCL benchmark: the dispatch inputs reach the task through its environment and are never spliced into the command. The task copies ccl_results/ out of its private checkout, so the publish and upload steps still find it, even on failure.
• acquire_gpus.sh is removed: nothing calls it any more. New GPU workflows should go through the queue.
• Reservation wait counts from when a task was queued. Before, the anti-starvation clock restarted each time a task launched. So in a job with several 8-GPU tasks, each one waited the full 5 minutes again before it could reserve the node, even after pending for an hour. On this PR's first perf run that added ~5 min per benchmark.

Trade-off: perf results are now one check, not four. Each benchmark is still reported separately in the job log, the ::error:: annotations and the step summary table, the same way #571 handled test directories. Running them back to back costs no parallelism: on a single 8-GPU host, two 8-GPU benchmarks could never run at the same time anyway.

Testing

• Queue: simulated locally with a temporary GPU state file and stub tasks: 2 GPU tasks, 8 GPU tasks, a failing task, and the CCL copy-out after a failure. Tasks got distinct GPUs and the 2-GPU pair ran together. A failure failed the step. The bitmap and held-GPU file ended at 0, and the temporary copies were removed. Bad ranks (9, 8;id) and a missing --install are rejected.
• Scripts: run_tests.sh and run_external_validation.sh were run with a stub container_exec.sh to check the --image and --gpus they pass.
• ruff and zizmor are clean.

Not yet run on GPUs. Perf and external validation run on this PR. Nightly and the CCL benchmark need a manual run from a branch in ROCm/iris.

🤖 Generated with Claude Code

[Copilot]

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open findings

• from_spec()no longer validates thatranks` is present and numeric before building the task name,… · New
• from_spec()no longer validates thatranks` is present and numeric before building the task name,… · New
• This script downloads and executes a remote test file. Since it is now centralized in a reusable… · New
• The --task ... "COMMAND" values embed long, multi-line shell strings with continuation… · New

What changed in this PR

Moves the remaining GPU-heavy CI workflows off the per-job bash GPU allocator and onto the per-task gpu_task_queue.py, adding support for queueing arbitrary GPU commands (not just test directories) to reduce runner-slot blocking and improve fairness via reservations.

Changes:

• Add --task RANKS NAME COMMAND to gpu_task_queue.py and allow mixing --task with existing TEST_DIR:RANKS tasks.
• Migrate performance regression, external validation, nightly Triton, and CCL benchmark workflows to use the queue instead of acquire_gpus.sh.
• Add run_external_validation.sh, update container/image handling (CONTAINER_IMAGE) in run_tests.sh, and remove acquire_gpus.sh.

File	Description
.github/​workflows/​iris-performance-regression-test.yml	Consolidates perf benchmarks into one queued job using --task for node reservation + reduced slot blocking.
.github/​workflows/​iris-nightly-triton-test.yml	Switches nightly Triton tests to queued per-directory tasks (1/2/4/8 ranks) using CONTAINER_IMAGE.
.github/​workflows/​iris-external-validation-test.yml	Runs Triton + Gluon external validations as two queued tasks in one job; passes repo/sha via env.
.github/​workflows/​iris-ccl-benchmark.yml	Runs CCL sweep via a queued task and copies results out of the task’s private checkout.
.github/​scripts/​run_tests.sh	Adds CONTAINER_IMAGE override and prints Triton version inside the container.
.github/​scripts/​run_new_examples.sh	Updates comment to reflect GPU_DEVICES now coming from gpu_task_queue.py.
.github/​scripts/​run_external_validation.sh	New script to run external validation via queued GPU tasks with correct --gpus propagation.
.github/​scripts/​release_gpus.sh	Updates comment to describe releasing GPUs held by gpu_task_queue.py after abnormal termination.
.github/​scripts/​gpu_task_queue.py	Adds --task, relaxes --install requirement to only positional tasks, and supports shell-command tasks.
.github/​scripts/​acquire_gpus.sh	Removed (no longer used after workflow migrations).

🧠 Review effort: Lite

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[mawad-amd]

Looks good. Thanks Nirvedh.

We initially had these actions that test multiple things however it was a pain to debug issues that came up in CI so we broke them down into smaller tasks. We can try again though.

Please confirm all CI is passing before merging. CI is backed up at the moment.

[nirvedhmeshram]

@mawad-amd yes the CI is passing, but looks like we should change the required labels before landing, I think you did that for the previous PR with admin access.

[nirvedhmeshram]

I was able to change the status checks so merging this so other PRs dont block on it.