Repository navigation
Make multi-host Iris workloads find the fabric or fail fast - #574
Merged
Merged
Conversation
- Get the AMD PCI bus ID from torch when the amdsmi Python bindings are missing. - Reuse the HIP runtime and libamd_smi torch already loaded (or the copy beside it) instead of mapping a second one; log when that overrides LD_LIBRARY_PATH. - Reject torch/vmem allocators on multi-host jobs; for vmem_chunked, require one shared fabric domain, and put each rank's discovery failure reason in the error. - Time out accept() and the rank handshake in the FD-passing mesh.
drprajap
requested review from
BKP,
artulab,
mawad-amd and
neoblizz
as code owners
October 8, 2026 04:17
github-actions
Bot
added
in-progress
artulab
approved these changes
Oct 8, 2026
artulab
left a comment
artulab
left a comment
Collaborator
There was a problem hiding this comment.
Changes look good to me.
mawad-amd
reviewed
Oct 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On two Helios trays (MI455X, one pod), IRIS_ALLOCATOR=vmem_chunked picked INTRA_NODE, built the
single-host FD-passing mesh and hung; IRIS_ALLOCATOR=vmem died with a bare HIP error code 1.
Nothing in the log pointed at fabric discovery.
Technical Details
Test Plan
New unit tests (CPU only): test_multihost_guard.py, test_platform_libs.py, and additions to
test_topology.py / test_drivers.py.
test_allocate_symmetric.py
(test_allocate_symmetric_returns_peer_bases: a 1 MiB heap rounds up to one 2 MiB granule;fails identically on main).
Test Result
Submission Checklist