Skip to content

fix(torch): honor handle streams in generated operators - #880

Open
voltjia wants to merge 3 commits into
masterfrom
fix/torch-backend-stream
Open

fix(torch): honor handle streams in generated operators#880
voltjia wants to merge 3 commits into
masterfrom
fix/torch-backend-stream

Conversation

@voltjia

@voltjia voltjia commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Installs the native stream stored in an InfiniOps Handle as PyTorch's current stream before generated ATen and linked Torch calls, then restores the previous device and stream.
  • Declares the shared C10<kDev> template in src/torch/c10.h and keeps each backend specialization in src/torch/<platform>/c10.h, following the existing platform-oriented C10 design without a separate stream bridge.
  • Makes generated wrappers, linked adapters, and the hand-written FlashAttention wrapper use the same platform C10 interface, while preserving PyTorch's current stream when the handle stream is null.
  • Adds the required vendor Torch headers, libraries, runtime paths, generator assertions, and current-stream versus explicit-handle regression coverage.

Motivation

Generated ATen backends received the InfiniOps handle stream through stream_, but they invoked PyTorch without installing that stream as the framework's current stream. PyTorch work could therefore run on a different stream from the caller-selected InfiniRT stream and violate the caller's ordering assumptions. Linked Torch adapters already modeled device-specific C10 behavior, so the shared implementation should retain that template structure and live under the common Torch backend rather than under linked.

No issue is linked.

Type of Change

  • feat - New feature, operator, or platform.
  • fix - Bug fix.
  • perf - Performance improvement without a behavioral change.
  • refactor - Code restructuring without a behavioral change.
  • test - Adding or fixing tests only.
  • docs - Documentation only.
  • build / ci - Build system or CI configuration.
  • chore - Tooling, formatting, or other non-code changes.
  • Breaking change.

Platforms Affected

  • CPU (WITH_CPU).
  • NVIDIA (WITH_NVIDIA).
  • Iluvatar (WITH_ILUVATAR).
  • MetaX (WITH_METAX).
  • Cambricon (WITH_CAMBRICON).
  • Moore (WITH_MOORE).
  • Ascend (WITH_ASCEND).
  • PyTorch C++ bindings (WITH_TORCH).
  • Build system, CMake, or CI.
  • Python bindings or user-facing API.

Smoke Test Result

Final validation SHA: f6e6f270df79fce687570ea1b42cbb58fe0ebc9b.

Local NVIDIA generated-plus-linked smoke build:
85 passed, 22 skipped in 9.77s.

Generator and architecture tests:
45 passed in 1.65s.

Generated and linked stream regressions:
4 passed in 2.47s.

GitHub Actions affected-platform matrices:
The main and shadow matrices passed for Iluvatar, MetaX, Cambricon, Moore, and Ascend. The final-SHA NVIDIA main and shadow jobs failed after their assigned runners went offline during execution. Neither job produced a failure log; the main job remained in the host-device unit-test step and the shadow job remained in the remote-task wait step.

Test Results on Supported Platforms

Platform Affected Build / Smoke Result Full Result / Notes
NVIDIA Yes Smoke passed locally. The generated-plus-linked build passed. Generated abs current-stream and handle-stream cases passed, and generated plus linked FlashAttention handle-stream cases passed for implementation slots 8 and 16. The preceding SHA with identical executable code passed the NVIDIA main matrix. The final-SHA main and shadow jobs failed after their assigned runners went offline during execution, without producing failure logs.
Iluvatar Yes Main and shadow CI passed. The CUDA-compatible C10 specialization uses InfiniRT's native stream type and c10::cuda::getStreamFromExternal.
MetaX Yes Main and shadow CI passed. The CUDA-compatible C10 specialization uses InfiniRT's native stream type and c10::cuda::getStreamFromExternal.
Cambricon Yes Main and shadow CI passed. The C10 specialization uses torch_mlu::getStreamFromExternal.
Moore Yes Main and shadow CI passed. The C10 specialization uses c10::musa::getStreamFromExternal from the explicitly linked libmusa_python.
Ascend Yes Main and shadow CI passed. The C10 specialization uses c10_npu::getStreamFromExternal when the installed headers provide NPUStreamUtils.h, and otherwise retains the current stream.
Local validation output.
Ruff 0.15.22 check: passed.
Ruff 0.15.22 format --check: passed.
Clang Format 21: passed.
Generator and architecture tests: 45 passed.
NVIDIA generated stream regression: 2 passed.
NVIDIA generated and linked FlashAttention stream regression: 2 passed.
NVIDIA smoke suite: 85 passed, 22 skipped.

Benchmark / Performance Impact

N/A. This is a stream-ordering correctness fix and an ownership refactor, and no performance benchmark was run.

Notes for Reviewers

InfiniRT remains the source of truth for each backend's native stream type through Runtime<kDev>::Stream. Vendor PyTorch APIs are used only by the corresponding C10<kDev> specialization to install that native stream into PyTorch's thread-local current-stream state.

src/torch/c10.h contains only the template declaration. Each platform owns its specialization under src/torch/<platform>/c10.h, matching the original C10 organization. The old src/linked/torch/c10.h, backend-local linked C10 headers, and the centralized src/torch/stream_.h bridge have been removed.

Generated and linked call sites hold an optional platform StreamGuard. They call C10<kDev>::GetStreamFromExternal only when InfiniRT supplies a non-null stream, so a null handle leaves PyTorch's selected current stream unchanged. The only platform-selection preprocessor conditions are the existing build guards needed to avoid including unavailable vendor headers; stream conversion itself is implemented by template specializations rather than macros.

NVIDIA, Iluvatar, and MetaX use c10::cuda::getStreamFromExternal. Moore uses c10::musa::getStreamFromExternal, Cambricon uses torch_mlu::getStreamFromExternal, and Ascend uses c10_npu::getStreamFromExternal when available.

Hygon is not claimed by this change because a validated PyTorch external-stream bridge for that backend is not available in the current test matrix. Its C10 specialization explicitly retains the current PyTorch stream.

@voltjia
voltjia requested a review from a team August 4, 2026 11:36
@voltjia
voltjia force-pushed the fix/torch-backend-stream branch 5 times, most recently from f457ca2 to f99e11e Compare August 5, 2026 10:15
@voltjia
voltjia force-pushed the fix/torch-backend-stream branch 4 times, most recently from cc00c1d to af8ae29 Compare August 9, 2026 10:48
@voltjia
voltjia force-pushed the fix/torch-backend-stream branch from af8ae29 to f6e6f27 Compare August 9, 2026 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant