Skip to content

refactor(cuda-core): share host handles and allocations across Tile and SIMT - #1603

Open
elibol wants to merge 8 commits into
NVIDIA:pull-request/1516from
elibol:refactor/shared-cuda-core
Open

elibol wants to merge 8 commits into
NVIDIA:pull-request/1516from
elibol:refactor/shared-cuda-core

Conversation

@elibol

@elibol elibol commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Tile and SIMT currently expose separate cuda-core contexts, streams, launch configurations, and allocation owners. A buffer moving between the models consequently needs raw-pointer adaptation, and the separate runtimes cannot coordinate lifetime or conflicting accesses.

This removes cuda_core::simt and gives both models the same host handles and allocation owner. Device identifies a physical GPU; Context owns or borrows the actual CUDA context. Typed buffers, contiguous sliced views, foreign owners, and Tile tensors share allocation identity and access tracking. Tensor::from_buffer, into_buffer, and as_buffer transfer or borrow storage without copying device memory. Ownership transfer rejects live typed aliases, preserving the boolean representation guarantees in #1516.

Safe submissions retain allocations and access leases until GPU completion, including after a borrowed launch future is dropped or forgotten. Shared references to device atomics are conservatively tracked as writable. Buffer and stream destructors retire work without waiting on the dropping thread; unknown completion retains resources instead of freeing active memory. The existing SIMT async launcher receives the small retention bridge needed to make this core migration safe before its separate consolidation.

Allocation uniqueness counts user-visible byte views independently of completion leases. This removes the retirement window where a decremented lease count still had a live Arc and made a unique buffer appear aliased. The same separation at the Tensor layer comes from #1516; both checks remain required for zero-copy ownership transfer.

Nihal's fixes preserve ordinary shared-read permissions on empty and peeled buffers and wait for retirement before checking reclaimed VRAM.

Consumer code, generated launchers, examples, templates, documentation, tests, and CI paths use the shared API. Tile numeric storage wrappers also implement DeviceCopy; DType and DeviceCopy remain independent traits.

Stack: depends on #1516. The base is its existing pull-request/1516 mirror so this PR shows only the core migration. Retarget to main after #1516 merges. The stacked follow-up #1604 removes cuda_async::simt.

Validation:

  • Full cuda-core suite: 111 passed; full cuda-async suite: 151 passed.
  • Selected Tile library/GPU/interop/submission-lifetime suites: 156 passed, including graph replay, offset preservation, alias rejection, and forgotten futures.
  • SIMT host/macro suites: 261 passed; cargo-oxide: 268 passed. Three additional GPU launch-lifetime regressions pass explicitly with --ignored.
  • Clippy with warnings denied for host workspace, Tile, and affected SIMT packages; strict cuda-core rustdoc; formatting, host-path and SPDX guards. The Tile workspace is also checked with every example feature enabled, including Candle CPU references.
  • Dependency checks pass for the host, SIMT, backend, device-only fixture, and Tile workspaces.

GPU validation used a single RTX 5090. Multi-GPU/NVLink cases self-skip on this machine; this does not claim multi-GPU coverage. Existing deliberately ignored tests remain ignored.

Closes #1460. Implements the physical-device/context distinction discussed in #1504.

Current follow-up validation: all 50 cuda-core unit tests and all nine targeted shared-allocation/retirement GPU tests pass, as does cuda-core Clippy with warnings denied. The new concurrency regression failed 3,308 uniqueness checks on the previous implementation and passes with the fix.

Current-main integration: Merged the updated #1516 prerequisite and migrated the new CuTe host consumers to shared LaunchConfig and the explicit event-flags API. Affected-package Clippy and all 16 CuTe example/model workspace Clippy checks pass. Fresh CI is running at 47042a8fae; the preceding head had all 63 active checks passing.

@elibol
elibol force-pushed the refactor/shared-cuda-core branch from d1ee216 to f0f61e2 Compare October 21, 2026 23:10
@nihalpasham

Copy link
Copy Markdown
Collaborator

Thanks @elibol, this looks good. I pushed a small commit (eec8776ff) for two minor things:

  • peel_elem and the num_bytes == 0 early returns in DeviceBuffer were leaving shared_access_writes = true (from relabel / from_raw_parts), which blocked concurrent read-only launches on peeled or empty buffers.
  • Added wait_for_retirement to vram_returns_to_baseline_after_buffer_cycles in device_buffer_leaks.rs so cuMemGetInfo_v2 doesn't race the background retirement thread.

@elibol

elibol commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator Author

@nihalpasham Thanks for the review and fixes. Your commit eec8776ff is preserved in the updated branch: peeled and empty buffers retain ordinary shared-read permissions, and the VRAM reclamation test waits for retirement before checking the baseline.

The follow-up work also fixes allocation uniqueness accounting by counting actual buffer views separately from GPU completion leases, and incorporates the corresponding tensor ownership fix from #1516. The regression fails on the earlier implementation and passes with the fix.

All of your feedback is incorporated. All 63 active CI checks pass at 36f9072030, including CodeQL; the targeted shared-allocation and retirement GPU tests also pass. This remains stacked on #1516.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants