Add prebuilt cuda-samples test suite to the base provider (New) by seankingyang · Pull Request #2797 · canonical/checkbox

seankingyang · 2026-08-26T09:57:41Z

Description

This PR adds a new CUDA test suite to the base provider (OEMQA-6843).
It runs every test program from NVIDIA's
cuda-samples repository as
its own Checkbox job. It works on any device with an NVIDIA GPU,
including Jetson/Tegra boards, and the same jobs work on Ubuntu
classic and Ubuntu Core.

How it works:

Checkbox does not download or build cuda-samples. The tree must
be cloned and built before the test run (by the test setup on
classic, or inside a snap on Ubuntu Core). A readiness job checks
this first and fails early when the tree is missing, not built, or
at the wrong version.
The version check: the cuda-samples checkout tag must be the newest
tag that is not newer than the CUDA version from nvidia-smi.
Example: CUDA 12.6 needs tag v12.5, because the next tag is
v13.0.
The script looks for the tree in this order: CUDA_SAMPLES_PATH,
then /opt/nvidia/cuda-samples (NVIDIA's Jetson layout), then
/snap/*/current/cuda-samples (Ubuntu Core snap).
A resource job lists every sample in the tree. One job is created
per sample, named cuda-sample/<samples-category>-<sample>. Both
old Makefile trees (<= v12.5) and new cmake trees (>= v13.0) work,
including the deeper Samples/8_Platform_Specific/Tegra/ folder.
Some samples can never run on some devices (for example: they need
two GPUs, or they need a display). These samples are listed in a
JSON exclude file, each with a written reason. exclude_jobs.json
is the default list; exclude_jobs.jetson.json adds the
Jetson-only excludes from NVIDIA's own runner, and a project picks
its list with CUDA_SAMPLES_EXCLUDE_FILE in the launcher. A list
can start from one shipped base list with an include key, so no
entries are copied twice (one include only, no chains, duplicates
are an error).
Excluded samples, and samples that were not built, still appear in
the results as skipped jobs with their reason. No sample ever
disappears silently.
Each sample runs as the normal user (no root) inside a temporary
sandbox folder. Because of this, tests also work when the tree is
owned by root, and a test run never changes the tree itself.
Every sample has a time limit (default 300 s,
CUDA_SAMPLES_TIMEOUT). Without it, some graphics samples run
forever on machines without a display.
36 unit tests cover the logic. One of them runs the job gate
expression from jobs.pxu through the real plainbox evaluator: the
parentheses in that expression are required, and the test fails if
someone removes them (plainbox would then evaluate each part
separately and the per-sample skip would stop working).

Resolved issues

Part of OEMQA-6843 (Jetson CUDA test migration). Related: OEMQA-6849
(rootless Tegra device-node setup), OEMQA-6850 (Ubuntu Core
cuda-samples snap).

Documentation

providers/base/units/cuda/README.md documents the prebuilt-only
contract, per-OS setup, environment variables, exclude-list format and
governance, and the sandbox/rootless design.

Tests

manage.py validate clean (no cuda-related issues in
--strict --deprecated); 36 unit tests green.
Jetson Orin (riverside, 202406-34151; CUDA 12.6, tree v12.5
Makefile era): 137 passed / 0 failed / 87 visible skips, 64 min.
Submission: https://certification.canonical.com/hardware/202406-34151/submission/507019/
GB300 (202604-38713; CUDA 13.2, tree v13.2 cmake era, in-source):
136 passed / 0 failed / 67 visible skips, 4 min 35 s.
Submission:
https://certification.canonical.com/hardware/202604-38713/submission/507000/
Full per-job outputs are in the C3 submissions above.

Run every NVIDIA cuda-samples test as its own Checkbox job on any device with an NVIDIA GPU (Jetson/Tegra included), with one identical entry point for Ubuntu classic and Ubuntu Core: - units/cuda/: category, has_nvidia_cuda manifest entry, resource job emitting one record per sample, prebuilt-only readiness job, template jobs named cuda-sample/<samples-category>-<sample>, and the cuda-samples-automated test plan. - bin/cuda_samples.py: list/readiness/run subcommands; prebuilt-only (never clones or builds); resolves the tree from CUDA_SAMPLES_PATH, /opt/nvidia/cuda-samples or a cuda-samples snap; verifies the checkout is at the newest tag <= the nvidia-smi CUDA version; runs each sample in a disposable sandbox (rootless-safe on root-owned trees, never modifies the provisioned tree) with a per-sample timeout and process-group kill. - data/cuda/exclude_jobs.json (+ JSON schema): universal excludes only (multi-GPU/P2P, cuDLA, headless EGL/GLES, non-self-contained tools); exclude_jobs.jetson.json adds the Jetson/L4T-specific excludes from NVIDIA's reference runner, selectable per project via CUDA_SAMPLES_EXCLUDE_FILE. Excluded and unbuilt samples still appear as visibly skipped jobs, never silently dropped. - tests/test_cuda_samples.py: 36 unit tests, including one that evaluates the template requires gate through the real plainbox evaluator. Validated on a Jetson Orin (CUDA 12.6, v12.5 Makefile-era tree, 137 pass / 0 fail / 87 visible skips) and a GB300 (CUDA 13.2, v13.2 cmake-era tree, 136 pass / 0 fail / 67 visible skips).

codecov · 2026-08-26T10:09:03Z

Codecov Report

❌ Patch coverage is 91.48936% with 24 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.13%. Comparing base (ba6965e) to head (77c752e).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
providers/base/bin/cuda_samples.py	91.48%	17 Missing and 7 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2797      +/-   ##
==========================================
+ Coverage   60.96%   61.13%   +0.17%     
==========================================
  Files         489      490       +1     
  Lines       49398    49680     +282     
  Branches     8834     8903      +69     
==========================================
+ Hits        30116    30374     +258     
- Misses      18356    18373      +17     
- Partials      926      933       +7

Flag	Coverage Δ
provider-base	`40.45% <91.48%> (+0.81%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

bladernr · 2026-08-26T13:06:47Z

Q: why is this being done in the base provider rather than the gpgpu provider? IMO we should not really have tests for the same thing in two different places, this just leads down the road to confusion around which provider to put which thing when new GPU tests come up. The gpgpu provider already has job definitions for a few prebuilt cuda samples that are provided in turn by the cuda-samples snap that @pedro-avalos maintains currently.

To be clear, I like this addition, I just think it needs to be done inside the gpgpu provider to maintain consistency. Plus I would welcome adding these extra samples tests to the gpu test suite for cert as well (this was one of the phase 2 ideas from before anyway)

pedro-avalos · 2026-08-26T13:54:49Z

I would say if this is more reliable, provides wider support, and is easier to maintain than the snap (which I find a pain to manage, especially as it lacks support for some architectures), then I would say, yeah this proposed prebuilt cuda-samples should replace the existing cuda-samples tests (of which there are two in gpgpu provider, I think: cuda-samples snap and I believe there is also one from partner engineering)

argparse's add_subparsers(required=True) needs Python 3.7 and made every cuda_samples.py invocation raise TypeError on the py35/py36 tox legs. Enforce the mandatory subcommand manually with parser.error (same usage message and exit code 2), and pin the behavior with a test.

cuDLA needs the Tegra Deep Learning Accelerator, so its samples are not a universal exclude: on non-Tegra platforms they are never built (visible 'not built' skip), and excluding them is a Jetson-context decision from NVIDIA's reference runner. Move the three cuDLA entries from the default exclude_jobs.json (now 12 universal entries, no Jetson wording) into exclude_jobs.jetson.json with a reason that explains the hardware dependency.

Copying the universal entries into every project list already caused one drift bug (a stale reason surviving in one copy). A list can now start from ONE shipped base list with an 'include' key, under strict rules that keep the effective list easy to trace: a bare file name resolved only in the provider's data/cuda directory, no nested include (an included file using 'include' is a loud error), and a duplicate name+category across the two files is an error instead of an overwrite. Unknown top-level keys are rejected too, so a typo like 'inclde' cannot be ignored silently. exclude_jobs.jetson.json now holds only the 12 Jetson-specific entries plus include: exclude_jobs.json (same 24 effective entries as before).

A category-less entry matches every category of that name, so a qualified entry for the same name on the other side of an include would load but never be consulted (the included entry sits first and wins) - a silently shadowed exclude reason. Treat such overlaps like duplicates: loud error naming both files. The same name in two different categories stays allowed, since those are two different samples.

Driver-API samples (matrixMulDrv, ptxjit, matrixMul_nvrtc, ...) load kernel .fatbin/.ptx files and nvrtc headers at runtime through sdkFindFilePath, which resolves them relative to the working directory. An out-of-source cmake build (cmake -B build) stages those sidecars beside the binary under build/, so the disposable sandbox (which mirrors only the sample's source directory) hid them and the 8 driver-API samples failed with 'sdkFindFilePath: file not found'. Overlay the binary's own directory into the sandbox run dir (skipping CMakeFiles) when it differs from the source directory. In-source make-era trees are unaffected: binary dir == source dir. Found running the CI-container-built cuda-samples-prebuilt snap tree on GB300 (OEMQA-6856); in-source device builds never hit this.

On Ubuntu Core there is no host CUDA toolkit, so dynamically linked samples (CUBLAS, cuSolver, cuFFT, nvrtc, jitLto, ...) fail at load time with 'libcublas.so: cannot open shared object file' when run from the checkbox agent: 30 of the runnable samples on a UC22 Jetson. The cuda-samples-prebuilt snap bundles the exact toolkit runtime its binaries were built against at usr/lib/host-gpu/cuda-runtime beside the tree. Append that directory to the sample's LD_LIBRARY_PATH — append, never prepend, so an existing loader path (the checkbox snap's runtime, a host toolkit) always wins. Plain classic trees have no such directory and are untouched. Found running the CI-built snap on the UC22 Jetson Core device (OEMQA-6856).

…h (Bugfix)" This reverts commit 540f5fb.

rickwu666666 · 2026-09-03T07:24:59Z

+ the jobs generated for them show up as visible skips instead of
+ silently disappearing from the run.
+estimated_duration: 5s
+imports: from com.canonical.plainbox import manifest


Can not use manifest in a resource job.

rickwu666666 · 2026-09-04T14:07:43Z

+
+
+def resolve_root(path=None):
+    """Return the first candidate containing a Samples/ directory."""


From cuda-samples v13.3, the folder rename from Samples to cpp.
https://github.com/NVIDIA/cuda-samples/tree/v13.3

rickwu666666 · 2026-09-04T14:19:28Z

I've test it on x86 platform with cuda_sapmles v13.3. And seems working.
https://certification.canonical.com/hardware/202607-39081/submission/509302/

rickwu666666 · 2026-09-04T14:32:29Z

+        )
+    except (OSError, subprocess.CalledProcessError) as exc:
+        raise SystemExit("Cannot run nvidia-smi: {}".format(exc))
+    match = re.search(r"CUDA Version:\s*(\d+)\.(\d+)", output)


While I'm using driver 610.57.04, the version pattern changed to CUDA UMD Version

airport@airport-Nuvo-7000-Series:~$ nvidia-smi Fri Sep 4 22:29:43 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 610.57.04 KMD Version: 610.57.04 CUDA UMD Version: 13.3 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX A2000 12GB On | 00000000:01:00.0 Off | Off | | 30% 54C P0 24W / 70W | 12MiB / 12282MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 2200 G /usr/bin/gnome-shell 2MiB | +-----------------------------------------------------------------------------------------+

seankingyang added 7 commits August 27, 2026 09:49

Revert "Put a snap tree's bundled CUDA runtime libs on the loader pat…

77c752e

…h (Bugfix)" This reverts commit 540f5fb.

rickwu666666 reviewed Sep 3, 2026

View reviewed changes

rickwu666666 reviewed Sep 4, 2026

View reviewed changes

Navigation Menu

Uh oh!

Add prebuilt cuda-samples test suite to the base provider (New) - #2797

Add prebuilt cuda-samples test suite to the base provider (New)#2797
seankingyang wants to merge 8 commits into
mainfrom
add-base-cuda-sample-tests

seankingyang commented Aug 26, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Aug 26, 2026 •

edited

Loading

Uh oh!

bladernr commented Aug 26, 2026 •

edited

Loading

Uh oh!

pedro-avalos commented Aug 26, 2026

Uh oh!

rickwu666666 Sep 3, 2026

Uh oh!

rickwu666666 Sep 4, 2026

Uh oh!

rickwu666666 commented Sep 4, 2026

Uh oh!

rickwu666666 Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants



		def resolve_root(path=None):
		"""Return the first candidate containing a Samples/ directory."""

Uh oh!

Conversation

seankingyang commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Resolved issues

Documentation

Tests

Uh oh!

codecov Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bladernr commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pedro-avalos commented Aug 26, 2026

Uh oh!

rickwu666666 Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

rickwu666666 Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

rickwu666666 commented Sep 4, 2026

Uh oh!

rickwu666666 Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

seankingyang commented Aug 26, 2026 •

edited

Loading

codecov Bot commented Aug 26, 2026 •

edited

Loading

bladernr commented Aug 26, 2026 •

edited

Loading