Snapshot/restore for GPU actors (suspend/resume with a live CUDA context)

[eliranw]

Goal

Let a GPU actor suspend and resume while the workload is holding a CUDA context. Follow-up to the "Snapshot/restore with GPU out of scope" note in #627.

What happens today (gVisor)

A Full-scope snapshot of an actor with a live CUDA context fails in runsc:

FATAL ERROR: checkpoint failed: checkpointing container "pause":
  encoding error: can't save with live nvproxy clients

The sandbox dies with it, so retries get cannot checkpoint container "pause" in state stopped. The actor ends up in STATUS_SUSPENDING, which you can't delete from, holding its worker. Same as #50.

Suggested approach (gVisor)

runsc already supports NVIDIA's cuda-checkpoint, which moves device memory to host and releases the GPU before saving. I tested it on a T4 with the runsc we pin (release-20260622.0): copied the binary into a running actor's rootfs and ran runsc checkpoint -cuda-checkpoint-path /cuda-checkpoint by hand. It went through, 301 MB of image files, no nvproxy error, workload still running after.

1. Get cuda-checkpoint into the actor. The flag wants a path inside the container, so it can ride the CDI mounts we already inject.
2. Pass -cuda-checkpoint-path in cmdCheckpoint (cmd/ateom-gvisor/runsc.go).
3. Restore needs nothing. gVisor replays the setting from the checkpoint image.

Micro-VM

Micro-VM actors don't have GPU support yet (#762), but the snapshot story should be settled before that lands, because it may constrain the design.

Need more exploration in this area.

Known costs / open issues

• The binary isn't in the driver or the toolkit. It's in-tree at NVIDIA/cuda-checkpoint under bin/x86_64_Linux/, no releases. About 6 KB, dynamically linked, so the actor image needs a loader. Same constraint that stopped distroless ateom from running nvidia-ctk.
• Golden snapshots are always Full, regardless of onPause/onCommit, since a data-only golden leaves nothing to restore the guest from. So a template holding a context at golden-snapshot time is affected either way.

Related: #627, #502, #762

[BenTheElder]

Does this have to have the same driver version, or just the same hardware? (Wondering for the scheduling APIs in #752)

[eliranw]

I think the comment on the other issue fits here better so:
After contacting the GPU snapshot/restore folks, the following should be same, at least for now, in cuda-checkpoint scenarios:

Driver version
GPU Type
GPU Count
Guest OS - should probably be same in general so linux kernel version is same.
Also you need to have enough memory allocated in the container to hold the cuda snapshot pre-suspend, this changes per device count and amount of device memory

[nybidari]

From gVisor size this approach makes sense! Once device passthrough is re-enabled and the -cuda-checkpoint-path flag is wired into cmdCheckpoint, live CUDA context suspend/resume should work as expected with gVisor.

We can also look into improving error handling in runsc/gVisor so that if a checkpoint is attempted with a live context (without cuda-checkpoint, or if the helper fails), it returns a non-fatal error to the caller rather than crashing the sandbox and leaving it in state stopped.