Skip to content

Support PyTorch 2.11.0 and PyTorch 2.13.0 with CUDA 13 - #1524

Merged
cbalioglu merged 4 commits into
mainfrom
torch213-cu130
Sep 8, 2026
Merged

cbalioglu merged 4 commits into
mainfrom
torch213-cu130

Conversation

@avidale

@avidale avidale commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Adds PyTorch 2.13.0 (cpu, cu130) to the support matrix, which enables co-installation with vllm==0.28.0 (pins torch==2.13.0 plus CUDA 13 runtime packages) and transformers~=5.16, removing friction in the downstream projects, such as Omnilingual.

Changes

Three things had to change to build against PyTorch 2.13:

  • PyTorch 2.12 and later set CMAKE_CXX_STANDARD 20, so fairseq2n is now compiled as C++20. This is a breaking change for source builds: GCC 11.3 or greater, or Clang 16 or greater, is now required. The CI images already ship gcc-toolset-14, so _build_wheel-linux.yaml selects it for PyTorch 2.12+ and keeps gcc-toolset-11 (GCC 11.2) for older versions.

  • Under C++20, fmt::format_string is checked at compile time, so runtime strings can no longer be passed as the format. sndfile.cc and sp_processor.cc now pass libsndfile and SentencePiece error text as an argument instead. This also fixes a latent bug: error text containing braces was previously interpreted as format syntax.

  • Also under C++20, the unqualified ssize() call in the memory bindings became ambiguous with std::ssize, because std::byte in the template argument pulls std into the ADL set. It is now qualified.

CUDA 13 dropped Volta, so the default CMAKE_CUDA_ARCHITECTURES is Turing rather than Volta when PyTorch is built against CUDA 13, and the wheel workflow selects architectures per variant.

Finally, the bundled zip third-party project compiles itself with -Werror, and GCC 14's new -Wcalloc-transposed-args makes that fatal, so -Werror is disabled for that target only.

Verified for torch 2.13.0+cu130 / py3.12 / x86_64: native suite 28/28, pytest --device cpu 1155 passed, pytest --device cuda:0 on an H100 1154 passed (the one failure, gemma3n test_batch_independence, is a pre-existing GPU float-tolerance issue: max abs diff 9.5e-07 with atol left at its 1e-8 default). Wheels built in the new manylinux_2_28 cu130 image audit clean at glibc 2.27.

Some additional changes were required to make the Github CI pass; they seem unrelated to the torch/cuda/transformers versions upgrade, and instead, chase the natural evolution of the other dependencies:

  • Change the Python version for the mypy check from 3.10 to 3.12, otherwise, the type stubs for numpy>=2.5, scipy, librosa are incompatible with the syntax
  • adapt to a refactoring of the wandb dependency: wandb.util.generate_id → from wandb.sdk.lib.runid import generate_id

Does your PR introduce any breaking changes? If yes, please list them:
List of all backwards-incompatible changes.

Check list:

  • Was the content of this PR discussed and approved via a GitHub issue? (no need for typos or documentation improvements)
  • Did you read the contributor guideline?
  • Did you make sure that your PR does only one thing instead of bundling different changes together?
  • Did you make sure to update the documentation with your changes? (if necessary)
  • Did you write any new necessary tests?
  • Did you verify new and existing tests pass locally with your changes?
  • Did you update the CHANGELOG? (no need for typos, documentation, or minor internal changes)

Adds PyTorch 2.13.0 (`cpu`, `cu130`) to the support matrix, which enables
co-installation with `vllm==0.28.0` (pins `torch==2.13.0` plus CUDA 13
runtime packages) and `transformers~=5.16`. Also completes the earlier
PyTorch 2.11.0 (`cpu`, `cu126`, `cu128`) work.

Three things had to change to build against PyTorch 2.13:

- PyTorch 2.12 and later set `CMAKE_CXX_STANDARD 20`, so fairseq2n is now
  compiled as C++20. This is a breaking change for source builds: GCC 11.3
  or greater, or Clang 16 or greater, is now required. The CI images already
  ship gcc-toolset-14, so `_build_wheel-linux.yaml` selects it for PyTorch
  2.12+ and keeps gcc-toolset-11 (GCC 11.2) for older versions.

- Under C++20, `fmt::format_string` is checked at compile time, so runtime
  strings can no longer be passed as the format. `sndfile.cc` and
  `sp_processor.cc` now pass libsndfile and SentencePiece error text as an
  argument instead. This also fixes a latent bug: error text containing
  braces was previously interpreted as format syntax.

- Also under C++20, the unqualified `ssize()` call in the memory bindings
  became ambiguous with `std::ssize`, because `std::byte` in the template
  argument pulls `std` into the ADL set. It is now qualified.

CUDA 13 dropped Volta, so the default `CMAKE_CUDA_ARCHITECTURES` is Turing
rather than Volta when PyTorch is built against CUDA 13, and the wheel
workflow selects architectures per variant.

Finally, the bundled `zip` third-party project compiles itself with
`-Werror`, and GCC 14's new `-Wcalloc-transposed-args` makes that fatal, so
`-Werror` is disabled for that target only.

Verified for torch 2.13.0+cu130 / py3.12 / x86_64: native suite 28/28,
`pytest --device cpu` 1155 passed, `pytest --device cuda:0` on an H100
1154 passed (the one failure, gemma3n `test_batch_independence`, is a
pre-existing GPU float-tolerance issue: max abs diff 9.5e-07 with
`atol` left at its 1e-8 default). Wheels built in the new
manylinux_2_28 cu130 image audit clean at glibc 2.27.
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 4, 2026
Comment thread .github/workflows/_build_wheels.yaml Dismissed
Comment thread .github/workflows/_build_wheels.yaml Dismissed
Comment thread .github/workflows/_build_wheels.yaml Dismissed
Comment thread .github/workflows/_publish.yaml Dismissed
Comment thread .github/workflows/_publish.yaml Dismissed
Comment thread .github/workflows/_publish.yaml Dismissed
@avidale
avidale marked this pull request as ready for review September 7, 2026 09:34
@avidale avidale changed the title [WiP] Support PyTorch 2.11.0 and PyTorch 2.13.0 with CUDA 13 Support PyTorch 2.11.0 and PyTorch 2.13.0 with CUDA 13 Sep 7, 2026
@cbalioglu
cbalioglu merged commit 7f06d6f into main Sep 8, 2026
25 checks passed
@cbalioglu
cbalioglu deleted the torch213-cu130 branch September 8, 2026 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants