[danieldk]

• Use NCCL version to match the version that Torch was built with.
• Ensure that a kernel can use NCCL by propagating it as a CUDA build input
• Add an example kernel to check in CI that NCCL can be built against.
• Provide mkNccl to easily construct NCCL derivations from Python wheels (hope to switch to CUDA manifests soon, but there are currently some clashing files).
• Support passing an overlay to genKernelFlakeOutputs.

[danieldk]

The latter change is much more general, but can be used by a kernel like DeepEP to pass a newer NCCL. For example:

{
  inputs.kernel-builder.url = "github:huggingface/kernels";

  outputs =
    { self, kernel-builder }:
    kernel-builder.lib.genKernelFlakeOutputs {
      inherit self;
      path = ./.;

      # This kernel needs NCCL 2.32.3 or newer, which is newer than the NCCL
      # that any of the supported Torch versions depend on.
      overlays = [
        (
          final: prev:
          let
            nccl = final.mkNccl {
              version = "2.32.3";
              wheels = {
                x86_64-linux.cuda13 = {
                  url = "https://files.pythonhosted.org/packages/5b/29/6b277e63c92d91f9cb4d1a3a554e148983de39d54baa652bb52c798af78e/nvidia_nccl_cu13-2.32.3-py3-none-manylinux_2_27_x86_64.whl";
                  sha256 = "1459723080ac889d73a26edfa3e04383a7928ab31ac8f0ec43b3ea9548b04ff3";
                };
                aarch64-linux.cuda13 = {
                  url = "https://files.pythonhosted.org/packages/58/0a/c29c302036a06d27dd732588f3fd8ee89b1f7b0087e38c668ae7be8ff7b2/nvidia_nccl_cu13-2.32.3-py3-none-manylinux_2_27_aarch64.whl";
                  sha256 = "a5bee92b2f4af218c109f8d221c3c9adcc752b94ae0ffcb0ed5abf9341a7724c";
                };
              };
            };
          in
          {
            pythonPackagesExtensions = prev.pythonPackagesExtensions ++ [
              (
                python-final: python-prev:
                prev.lib.mapAttrs (_: torch: torch.override { inherit nccl; }) (
                  prev.lib.filterAttrs (name: _: prev.lib.hasPrefix "torch-bin_" name) python-prev
                )
              )
            ];
          }
        )
      ];
    };
}

[github-actions]

Coverage report — kernels/

Measured on: Python 3.10 / Torch 2.13.0.
Other CI configurations are not included in this number.
Hardware-gated code paths (ROCm/XPU/NPU/Darwin/Windows) are excluded or unreachable on the Linux+CUDA runner.

Total coverage: 88.5% — threshold: 80% — ✅

Per-file breakdown

Name	Stmts	Miss	Cover	Missing
src/kernels/__init__.py	14	0	100%
src/kernels/_system.py	6	1	83%	10
src/kernels/_versions.py	130	14	89%	53, 59-60, 63-64, 102, 165-170, 199, 219
src/kernels/archs.py	56	1	98%	94
src/kernels/backends.py	215	63	71%	42, 46, 50-53, 70, 92, 110, 119, 123, 127-129, 150, 159, 163, 167-169, 190, 201, 203, 210-213, 222, 226, 230-255, 263, 286-306
src/kernels/compat.py	9	1	89%	5
src/kernels/deps.py	70	1	99%	56
src/kernels/digest.py	58	5	91%	71-73, 136-137
src/kernels/hf_hub.py	62	2	97%	21, 23
src/kernels/importer.py	74	5	93%	158, 162, 165, 179-180
src/kernels/install.py	21	7	67%	76-100
src/kernels/layer/__init__.py	7	0	100%
src/kernels/layer/_interval_tree.py	103	4	96%	23, 52, 147, 150
src/kernels/layer/device.py	48	12	75%	42, 47-49, 91, 96-98, 101, 149, 152, 156
src/kernels/layer/func.py	93	5	95%	94, 119, 195, 315, 390
src/kernels/layer/globals.py	5	0	100%
src/kernels/layer/inspect.py	27	0	100%
src/kernels/layer/kernelize.py	83	7	92%	306, 314-315, 321, 325, 341-343
src/kernels/layer/layer.py	320	14	96%	212, 259, 447, 606-607, 628, 636, 647, 688, 692, 705, 760, 801, 876
src/kernels/layer/mode.py	14	0	100%
src/kernels/layer/repos.py	136	36	74%	32, 35-42, 73, 86, 90, 99-100, 103, 106-109, 116-117, 120, 123-126, 133-134, 137, 140-143, 150-151, 154, 157-160, 235
src/kernels/load.py	71	2	97%	338, 378
src/kernels/locking.py	89	64	28%	35-83, 91-98, 102-125, 137, 152-159, 165-175, 179-186
src/kernels/python_deps.py	58	6	90%	59-60, 64-65, 101, 104
src/kernels/resolver.py	162	1	99%	248
src/kernels/status.py	50	2	96%	25, 79
src/kernels/validate.py	104	7	93%	9, 101, 172, 195-196, 232-233
src/kernels/variants.py	278	17	94%	65, 96, 117, 147, 256-257, 299-302, 304, 388-394, 400-406, 455-461
src/kernels/verify.py	102	6	94%	44, 176-178, 266-267
TOTAL	2465	283	89%

Updated by the Test kernels workflow on commit 5821c964f7a7b0a557076dbe9d7704b5592608dd.

[drbh]

lgtm!

[sayakpaul]

Shouldn't this also exist in https://github.com/huggingface/kernels-test so that we also properly sign this in case an upload is needed? @danieldk