Skip to content

Fix workspace allocation and reuse in matmul_all_reduce - #581

Draft
MashrafeeAryan wants to merge 5 commits into
ROCm:mainfrom
MashrafeeAryan:MashrafeeAryan/fix-matmul-all-reduce-workspace
Draft

MashrafeeAryan wants to merge 5 commits into
ROCm:mainfrom
MashrafeeAryan:MashrafeeAryan/fix-matmul-all-reduce-workspace

Conversation

@MashrafeeAryan

Copy link
Copy Markdown

Rationale for this change

When one or more GPUs are used, if one GPU needed new buffers while another skipped
allocation, the operation could hang and not proced,

Fixes #464.

What changes are included in this PR?

  • Separate buffer allocation from preparation for each calculation.
  • Make all GPU workers decide together whether they need new buffers. If any workers need them, all workers allocate memory
  • For one-shot and two-shot methods, give each calculation a number. Use that number to track when its results are ready and when all workers have finished using them. This allows buffers to be reused safely without clearing them every time.
  • Update benchmarks to let the operation manage preparation.
  • Add GPU tests for collective allocation and repeated calculations.

Are these changes tested?

GPU tests and performance measurements have not been run because
no AMD GPUs are available locally.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

matmul_all_reduce_preamble mixes collective allocation with per-call preparation

1 participant