[pull] main from containerd:main - #470
Merged
Merged
Conversation
The snapshot garbage collector holds the snapshotter write lock while calling the underlying Snapshotter.Remove. cleanupSnapshotter also detaches cancellation (context.WithoutCancel), so a Remove that never returns - e.g. a proxy snapshotter whose RPC connection has wedged - holds the write lock forever. Every subsequent snapshot operation on the node then blocks behind the lock, including CRI RunPodSandbox, turning the node into a permanent scheduling black hole while it still reports Ready. Observed in production with a proxy snapshotter (soci) that stopped answering after a file-descriptor exhaustion storm: the GC goroutine sat in WaitGroup.Wait for 4.5 hours with the Remove RPC parked in waitOnHeader, and 17 CRI operations queued behind the metadata lock. Bound each GC-path Remove with a timeout registered via pkg/timeout (io.containerd.timeout.gc.snapshotter.remove, default 30m, operator-tunable through config.toml like the other timeout keys). A timed-out Remove returns DeadlineExceeded, which is not a FailedPrecondition, so it propagates out of pruneBranch and ends the pass: a globally wedged snapshotter costs one timeout per GC cycle rather than one per orphaned snapshot, the deferred Cleanup is skipped because the pass failed, and the snapshots left behind stay orphaned for the next pass. Signed-off-by: Nahum Litvin <[email protected]>
metadata: bound snapshotter Remove during garbage collection
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )