Skip to content

Move kmod-6.18 nvidia drivers to use overlayfs at boot - #536

Open
vigh-m wants to merge 3 commits into
bottlerocket-os:developfrom
vigh-m:nvidia-multi-driver/kernel-kit
Open

Move kmod-6.18 nvidia drivers to use overlayfs at boot#536
vigh-m wants to merge 3 commits into
bottlerocket-os:developfrom
vigh-m:nvidia-multi-driver/kernel-kit

Conversation

@vigh-m

@vigh-m vigh-m commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Description of changes:
Enables the NVIDIA lts (r580) and fb (r595) driver branches to be co-installed on a single Bottlerocket image, with exactly one branch activated at boot. Single-driver images continue to work unchanged.

Both branches previously installed their libraries, binaries, module objects and configs directly onto the canonical system paths (/usr/lib64, /usr/bin, /usr/share/nvidia, /etc/nvidia, ...). This PR moves each branch's files under a per-branch storage root and overlays the selected branch onto the canonical paths at boot.

  • Each branch installs into /usr/nvidia/<branch>/ (lts or fb).
  • Per-branch, oneshot services mount overlays that make the storage root's contents appear at the canonical paths:
    • nvidia-<branch>-overlay-driver.service: lib64, bin, share-nvidia, share-vulkan, share-glvnd, share-egl.
    • nvidia-<branch>-overlay-config.service: etc-nvidia -> /etc/nvidia, etc-nvidia-imex -> /etc/nvidia-imex.
  • The /usr/lib64 overlay would otherwise shadow the writable /usr/lib/modules mount, so the base module tree is bound to a /run mount point and re-layered as a lowerdir with a writable upper under /var/lib/kernel-modules/.nvidia.
  • driverdog link/load/copy unit filenames are namespaced per branch (nvidia-lts-* / nvidia-fb-*) so both branches' units coexist without collision. driverdog config tomls are also branch-named and installed normally into the flat /etc/drivers.

New driver file layout

Each branch now lives under its own root, e.g.:

  [root@admin]# tree /.bottlerocket/rootfs/usr/nvidia/ -d
  /.bottlerocket/rootfs/usr/nvidia/
  `-- fb
      |-- bin
      |-- etc-nvidia
      |-- etc-nvidia-imex
      |-- lib64
      |   `-- tesla
      |-- share-egl
      |   `-- egl_external_platform.d
      |-- share-glvnd
      |   `-- egl_vendor.d
      |-- share-nvidia
      |   |-- grid
      |   |   `-- drivers
      |   |-- open-gpu
      |   |   `-- drivers
      |   `-- tesla
      |       |-- module-objects.d
      |       `-- nvswitch
      `-- share-vulkan
          |-- icd.d
          `-- implicit_layer.d

Renames

To keep the projected roots unambiguous under the storage root, the
packaged trees were renamed:

Before After Overlayed onto
share/ share-nvidia/ /usr/share/nvidia
etc/ (nvidia) etc-nvidia/ /etc/nvidia
etc/ (imex) etc-nvidia-imex/ /etc/nvidia-imex

Backwards compatibility

  • A single-driver image installs exactly one branch; its overlay services project that branch onto the canonical paths, giving the same runtime layout as before.
  • co-install guard: add a Provides and a Conflicts that guards lts+fb co-install unless kmod-nvidia-multi is present. This is to prevent any attempts to install both drivers without the right logic to pick the right driver at boot

Testing

  • Single-driver lts (r580) image: boots, driver loads, nvidia-smi works
  • Single-driver fb (r595) image: boots, driver loads, nvidia-smi works
  • Overlay mounts present after boot (lib64/bin/share-nvidia/etc-nvidia)
  • Nvidia Smoke test workload on g4dn, g5, g6, g7(595 only)
  • MIG and EFA testing (580 and 595)

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

vigh-m added 3 commits August 26, 2026 16:33
Install the driver's libraries, binaries, kernel-module objects,
driverdog tomls and /etc configs under a per-branch storage root
(/usr/nvidia/lts) instead of directly onto the canonical system paths,
and overlay them into place at bootime.

- Add nvidia-lts-overlay-driver.service: a oneshot that overlays lib64,
  bin, share-nvidia, share-vulkan, share-glvnd and share-egl from the
  storage root onto their canonical /usr and /usr/share paths
- Add nvidia-lts-overlay-config.service: overlay etc-nvidia onto
  /etc/nvidia and etc-nvidia-imex onto /etc/nvidia-imex
- Namespace the driverdog module sets and the link/load/copy unit
  filenames under the branch so the overlay wiring is unambiguous

Signed-off-by: Vighnesh Maheshwari <[email protected]>
Install the driver's libraries, binaries, kernel-module objects,
driverdog tomls and /etc configs under a per-branch storage root
(/usr/nvidia/fb) instead of directly onto the canonical system paths,
and overlay them into place at bootime.

- Add nvidia-fb-overlay-driver.service: a oneshot that overlays lib64,
  bin, share-nvidia, share-vulkan, share-glvnd and share-egl from the
  storage root onto their canonical /usr and /usr/share paths
- Add nvidia-fb-overlay-config.service: overlay etc-nvidia onto
  /etc/nvidia and etc-nvidia-imex onto /etc/nvidia-imex
- Namespace the driverdog module sets and the link/load/copy unit
  filenames under the branch so the overlay wiring is unambiguous

Signed-off-by: Vighnesh Maheshwari <[email protected]>
Adds the full IMEX support and a default-channel modprobe config

- Add nvidia-imex.service, its config.cfg and tmpfiles to the imex
  subpackage.

Signed-off-by: Vighnesh Maheshwari <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant