Repository navigation
SDMA is Triton-only, and no collective uses it #593
Description
SDMA is Triton-only, and no collective uses it
Checked on main @ 6462f028.
Where SDMA exists today
iris/mem/triton/ops.py exposes copy_engine_ctx / use_copy_engine on four
entry points:
| op | def at |
|---|---|
put |
280 |
atomic_add |
511 |
atomic_cas |
638 |
quiet |
964 |
Where it does not
Gluon has no SDMA surface. iris/mem/gluon/context.py mirrors the Triton RMA
set — load, store, get, put, copy, and nine atomics — but none of them
take a copy-engine argument, and sdma appears nowhere under iris/mem/gluon/.
No collective uses SDMA on either backend. grep -r "sdma\|copy_engine" iris/ccl/ returns nothing. Every kernel under iris/ccl/triton/ and
iris/ccl/gluon/ moves bytes with CU loads and stores.
So SDMA is available to someone hand-writing a Triton kernel against the RMA
primitives, and to nobody else. ctx.ccl.* cannot reach it.
Note the shape this leaves us in: the Gluon backend does have TDM
(iris/ccl/gluon/all_gather_tdm.py, #585, gfx1250), so Gluon got the newer
data-movement engine but not MI300-series copy engines, and the collectives got
neither.
Scope
Two pieces, in order:
- Gluon copy-engine surface — add
copy_engine_ctx/use_copy_engineto
IrisDeviceCtx.put(and the atomics that have it on the Triton side), matching
the Triton signatures so kernels can move between backends. - An SDMA collective — one collective, one backend, as a proof point. All-gather
is the natural first: the transfer is a contiguous per-destination copy, which
is exactly thecontiguous_copy=Truepathputalready supports.
Work
- Mirror
copy_engine_ctx/use_copy_engineiniris/mem/gluon/context.py - Add a copy-engine all-gather variant behind
all_gather_variant="sdma" - Decide whether
Configgrows an engine axis or SDMA stays a variant name - Correctness vs
torch.distributedon 2/4/8 ranks before timing - Benchmark vs the CU-driven variants through
benchmark/ccl/bench_all_gather.py,
which already has acomm_smsaxis — the interesting number is CU count at
equal bandwidth, not bandwidth at equal CU count
Related
#568 (engines= / copy-engine API), #573 (triggered SDMA), #585 (TDM all-gather).
Activity
ariscript commentedon Oct 16, 2026
I was planning on adding Gluon copy-engine features and a triggered all-gather collective after #573 merges. Mirroring the Triton API for Gluon should be fairly straightforward.
ariscript commentedon Oct 26, 2026
#605 mirrors the Triton device views for Gluon