update-ldcache hook: exec ldconfig from a sealed memfd - #2059
Open
klueska wants to merge 1 commit into
Open
Conversation
klueska
requested review from
cdesiniotis,
henry118 and
tariq1890
as code owners
September 9, 2026 11:57
klueska
force-pushed
the
memfd-execveat-ldcache-hook
branch
from
September 9, 2026 11:58
f01c290 to
39412ef
Compare
Coverage Report for CI Build 34443089629Coverage increased (+0.01%) to 44.03%Details
Uncovered Changes
Coverage Regressions6 previously-covered lines in 3 files lost coverage.
Coverage Stats
💛 - Coveralls |
This was referenced Sep 9, 2026
klueska
added a commit
to klueska/ray
that referenced
this pull request
Sep 9, 2026
…or GPU sandboxes
update-ldcache stays disabled under gVisor (nvidia-ctk's hook needs to
bind-mount /proc, which doesn't work from gVisor's gofer_mount hook
execution), so CDI-mounted driver libraries never get their ldcache
updated. Harmless on Debian/Ubuntu images, where the mounted path
already sits on ld.so's default search path, but breaks dynamic
linking (e.g. nvidia-smi's dlopen("libnvidia-ml.so.1")) on RHEL/Alpine
images.
nvidia-ctk always sets NVIDIA_CTK_LIBCUDA_DIR to those same
directories in its CDI edits. image_manager._apply_gpu_cdi_edits now
prepends that value onto LD_LIBRARY_PATH by name/reference right after
merging the CDI edits, purely as a workaround for the disabled hook --
marked with a TODO to remove once
NVIDIA/nvidia-container-toolkit#2059 lands and
is backported to a 1.18.x release, at which point re-enabling the hook
makes this unnecessary. Unlike an ldconfig-based fix this needs no
writable /etc/ld.so.cache, so it works regardless of the sandbox's
readonly setting.
Verified against a real RHEL UBI9-minimal sandbox, including the case
where the sandbox's own image/env= already sets LD_LIBRARY_PATH (kept,
not clobbered). Adds test_sandbox_gpu_nvidia_smi_runs_on_non_debian_image
to cover it.
Signed-off-by: Kevin Klues <[email protected]>
klueska
added a commit
to klueska/ray
that referenced
this pull request
Sep 9, 2026
…or GPU sandboxes
update-ldcache stays disabled under gVisor (nvidia-ctk's hook needs to
bind-mount /proc, which doesn't work from gVisor's gofer_mount hook
execution), so CDI-mounted driver libraries never get their ldcache
updated. Harmless on Debian/Ubuntu images, where the mounted path
already sits on ld.so's default search path, but breaks dynamic
linking (e.g. nvidia-smi's dlopen("libnvidia-ml.so.1")) on RHEL/Alpine
images.
nvidia-ctk always sets NVIDIA_CTK_LIBCUDA_DIR to those same
directories in its CDI edits. image_manager._apply_gpu_cdi_edits now
prepends that value onto LD_LIBRARY_PATH by name/reference right after
merging the CDI edits, purely as a workaround for the disabled hook --
marked with a TODO to remove once
NVIDIA/nvidia-container-toolkit#2059 lands and
is backported to a 1.18.x release, at which point re-enabling the hook
makes this unnecessary. Unlike an ldconfig-based fix this needs no
writable /etc/ld.so.cache, so it works regardless of the sandbox's
readonly setting.
Verified against a real RHEL UBI9-minimal sandbox, including the case
where the sandbox's own image/env= already sets LD_LIBRARY_PATH (kept,
not clobbered). Adds test_sandbox_gpu_nvidia_smi_runs_on_non_debian_image
to cover it.
Signed-off-by: Kevin Klues <[email protected]>
UpdateLDCache used to mount a fresh /proc inside the container root, just so it could bind-mount the host ldconfig binary in through a TOCTOU-safe /proc/self/fd/<n> target and then exec it by that same path once pivoted. This breaks under gVisor. Its createContainer hook doesn't have permission to mount a new /proc inside the container, so mountProc fails with EPERM. This constitutes a real regression, because this hook worked fine before the changes described above were introduced. To address this, UpdateLDCache now clones the host ldconfig binary into a sealed, anonymous memfd (via libcontainer/exeseal.CloneBinary) before touching any namespace or pivoting, then execs it by descriptor with execveat(fd, "", ..., AT_EMPTY_PATH). A memfd has no path and no mount-namespace membership, so it comes through pivot_root untouched. We don't need to bind-mount anything into the container, and we don't need to mount /proc anywhere. This is also more isolated than before. Since ldconfig has no use for /proc or /sys, prepareRoot now masks both with an empty tmpfs before pivoting, if they're present in the container root, instead of leaving a real working procfs mounted there and trusting ldconfig never reads it. We verified all of this against real GPU hardware through gVisor's runsc runtime, as well as standard runc. `docker run --runtime=runsc --device nvidia.com/gpu=0` against a non-Debian image resolves the driver libraries and runs nvidia-smi correctly, confirming the hook and the /proc/sys masking both work under gVisor's restricted- namespace gofer process. Signed-off-by: Kevin Klues <[email protected]>
klueska
force-pushed
the
memfd-execveat-ldcache-hook
branch
from
September 10, 2026 05:56
39412ef to
9353dc4
Compare
klueska
added a commit
to klueska/ray
that referenced
this pull request
Sep 10, 2026
…or GPU sandboxes
update-ldcache stays disabled under gVisor (nvidia-ctk's hook needs to
bind-mount /proc, which doesn't work from gVisor's gofer_mount hook
execution), so CDI-mounted driver libraries never get their ldcache
updated. Harmless on Debian/Ubuntu images, where the mounted path
already sits on ld.so's default search path, but breaks dynamic
linking (e.g. nvidia-smi's dlopen("libnvidia-ml.so.1")) on RHEL/Alpine
images.
nvidia-ctk always sets NVIDIA_CTK_LIBCUDA_DIR to those same
directories in its CDI edits. image_manager._apply_gpu_cdi_edits now
prepends that value onto LD_LIBRARY_PATH by name/reference right after
merging the CDI edits, purely as a workaround for the disabled hook --
marked with a TODO to remove once
NVIDIA/nvidia-container-toolkit#2059 lands and
is backported to a 1.18.x release, at which point re-enabling the hook
makes this unnecessary. Unlike an ldconfig-based fix this needs no
writable /etc/ld.so.cache, so it works regardless of the sandbox's
readonly setting.
Verified against a real RHEL UBI9-minimal sandbox, including the case
where the sandbox's own image/env= already sets LD_LIBRARY_PATH (kept,
not clobbered). Adds test_sandbox_gpu_nvidia_smi_runs_on_non_debian_image
to cover it.
Signed-off-by: Kevin Klues <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
UpdateLDCacheused to mount a fresh/procinside the container root, just so it could bind-mount the hostldconfigbinary in through a TOCTOU-safe/proc/self/fd/<n>target and then exec it by that same path once pivoted.This breaks under gVisor. Its
createContainerhook doesn't have permission to mount a new/procinside its sandbox, somountProcfails withEPERM. This constitutes a real regression, because this hook worked fine before the changes described above were introduced.To address this,
UpdateLDCachenow clones the hostldconfigbinary into a sealed, anonymousmemfd(vialibcontainer/exeseal.CloneBinary) before touching any namespace or pivoting, then execs it by descriptor withexecveat(fd, "", ..., AT_EMPTY_PATH). Amemfdhas no path and no mount-namespace membership, so it comes throughpivot_rootuntouched. We don't need to bind-mount anything into the container, and we don't need to mount/procanywhere.This is also more isolated than before. Since
ldconfighas no use for/procor/sys,prepareRootnow masks both with an emptytmpfsbefore pivoting, if they're present in the container root, instead of leaving a real workingprocfsmounted there and trustingldconfignever reads it.Test plan
go build ./internal/ldconfig/... ./cmd/nvidia-cdi-hook/...go vet ./internal/ldconfig/...go test ./internal/ldconfig/...runscruntime, as well as standardrunc:docker run --runtime=runsc --device nvidia.com/gpu=0against a non-Debian image resolves the driver libraries and runsnvidia-smicorrectly, confirming the hook and the/proc//sysmasking both work under gVisor's restricted-namespace gofer process.