Skip to content

nvca: capture claim outlives a dead owner pod and pins the function version Capturing #2154

Description

@balajinvda

Problem

The NvSnap capture-once claim in NvSnapFunctionState (captureOwner, captureLeaseExpiry, ~50 min TTL) is released only on a terminal outcome or lease expiry. A pod that dies mid-capture (evicted, replaced by a redeploy, its node agent restarted under it) leaves the function version in Capturing with nobody working on it. New pods of that version see the live claim and stand down for the rest of the lease, so the function stays cold for up to 50 minutes.

Observed on a dev cluster: the nvsnap agent DaemonSet rolled while the reconciler's checkpoint POST was in flight, the pod was replaced, and the replacement could not claim a capture until the lease expired.

Expected

A live claim whose owner pod no longer exists (or is terminating) is stealable at once. Unknown liveness must keep the claim so two pods never capture concurrently.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingreleased

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions