Skip to content

SDMA is Triton-only, and no collective uses it #593

Open

Description

@mawad-amd

SDMA is Triton-only, and no collective uses it

Checked on main @ 6462f028.

Where SDMA exists today

iris/mem/triton/ops.py exposes copy_engine_ctx / use_copy_engine on four
entry points:

op def at
put 280
atomic_add 511
atomic_cas 638
quiet 964

Where it does not

Gluon has no SDMA surface. iris/mem/gluon/context.py mirrors the Triton RMA
set — load, store, get, put, copy, and nine atomics — but none of them
take a copy-engine argument, and sdma appears nowhere under iris/mem/gluon/.

No collective uses SDMA on either backend. grep -r "sdma\|copy_engine" iris/ccl/ returns nothing. Every kernel under iris/ccl/triton/ and
iris/ccl/gluon/ moves bytes with CU loads and stores.

So SDMA is available to someone hand-writing a Triton kernel against the RMA
primitives, and to nobody else. ctx.ccl.* cannot reach it.

Note the shape this leaves us in: the Gluon backend does have TDM
(iris/ccl/gluon/all_gather_tdm.py, #585, gfx1250), so Gluon got the newer
data-movement engine but not MI300-series copy engines, and the collectives got
neither.

Scope

Two pieces, in order:

  1. Gluon copy-engine surface — add copy_engine_ctx / use_copy_engine to
    IrisDeviceCtx.put (and the atomics that have it on the Triton side), matching
    the Triton signatures so kernels can move between backends.
  2. An SDMA collective — one collective, one backend, as a proof point. All-gather
    is the natural first: the transfer is a contiguous per-destination copy, which
    is exactly the contiguous_copy=True path put already supports.

Work

  • Mirror copy_engine_ctx / use_copy_engine in iris/mem/gluon/context.py
  • Add a copy-engine all-gather variant behind all_gather_variant="sdma"
  • Decide whether Config grows an engine axis or SDMA stays a variant name
  • Correctness vs torch.distributed on 2/4/8 ranks before timing
  • Benchmark vs the CU-driven variants through benchmark/ccl/bench_all_gather.py,
    which already has a comm_sms axis — the interesting number is CU count at
    equal bandwidth, not bandwidth at equal CU count

Related

#568 (engines= / copy-engine API), #573 (triggered SDMA), #585 (TDM all-gather).

Activity

  1. ariscript commented on Oct 16, 2026

    @ariscript
    Member

    I was planning on adding Gluon copy-engine features and a triggered all-gather collective after #573 merges. Mirroring the Triton API for Gluon should be fairly straightforward.

  2. self-assigned this
    on Oct 16, 2026
  3. ariscript commented on Oct 26, 2026

    @ariscript
    Member

    #605 mirrors the Triton device views for Gluon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestgluonhelp wantedExtra attention is neededirisIris project issue

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions