Skip to content

cache/s3: fix a panic, an off-by-one and a discarded abort in the touch path - #7124

Closed
ernetas wants to merge 1 commit into
moby:masterfrom
ernetas:s3-touch-path-fixes
Closed

cache/s3: fix a panic, an off-by-one and a discarded abort in the touch path#7124
ernetas wants to merge 1 commit into
moby:masterfrom
ernetas:s3-touch-path-fixes

Conversation

@ernetas

@ernetas ernetas commented Sep 5, 2026

Copy link
Copy Markdown

When a blob is already in the bucket the exporter refreshes its timestamp instead of uploading it again. There are three separate problems on that path.

A nil dereference takes the daemon down

exists returns head.ContentLength, which is a *int64. The SDK leaves it nil when the response has no Content-Length header. touch then does:

if *size < maxCopyObjectSize {

The SDK models the field as optional, so nothing guarantees it is set. When it is not, this is a nil dereference, and an export that should have failed with an error panics the daemon instead.

I have not reproduced a nil from a specific endpoint. The argument here is only that the code dereferences a pointer the SDK says may be absent, and that the cost of being wrong is the whole daemon.

Check it at the call site and pass the size by value, so touch cannot be handed a nil in the first place.

The copy range can name a byte past the end of the object

buildCopySourceRange clamps like this:

end := start + maxCopyObjectSize - 1
if end > objectSize {
	end = objectSize - 1
}

A copy source range is inclusive. For an object of objectSize bytes the last byte you may name is objectSize-1. So an end equal to objectSize is already one too far, and > lets it through.

You need the remaining bytes to be exactly one part short of the part size to hit it, which is why nobody has. The bound should be >=.

A failed multipart upload is never actually aborted

The cleanup runs on the context that just failed:

if err != nil {
	s3Client.AbortMultipartUpload(ctx, &abortIn)
}

If the export was cancelled, or the context timed out, that context is already dead and the abort fails immediately. The result is discarded, so there is no log line and no error either.

An incomplete multipart upload keeps the parts that were uploaded. They do not show up in a bucket listing, but they are billed until something removes them. If you do not have a lifecycle rule for AbortIncompleteMultipartUpload, they stay forever.

Abort on a context detached from the failed one, with its own 30 second timeout, and log it if that fails too.

Tests

Adds a table test for buildCopySourceRange. Every case also asserts that the range never names a byte at or past objectSize, which is the property that was broken.

The "remainder one byte short of the part size" case fails on the current bound.

The other two fixes are on paths that need a real bucket, so they are not covered here.

Notes

This is independent of #7123 and #7121. It only touches the touch path.

…ch path

When a blob is already in the bucket the exporter refreshes its timestamp
instead of uploading it again. Three things are wrong on that path.

HeadObject's ContentLength was dereferenced without a check. It is a
*int64 and the SDK leaves it nil when the response carries no
Content-Length header, which several S3-compatible endpoints do. That
panicked the daemon rather than failing the export. Check it at the call
site and pass the size by value, so touch cannot be handed a nil.

buildCopySourceRange clamped with end > objectSize. A copy source range is
inclusive, so the last byte it may name is objectSize-1, and an end equal
to objectSize addresses one byte past the object. It is reachable when the
remaining bytes are exactly one part short of the part size.

The abort of a failed multipart upload ran on the context that had just
failed, which is usually already cancelled, so the abort failed too. Its
error was discarded, so nothing said so. An upload left incomplete keeps
its parts, and the bucket keeps charging for them until a lifecycle rule
removes them. Abort on a detached context with its own timeout and log a
failure.

Add a test for the copy source range. The one byte short case fails on the
old bound.

Signed-off-by: Ernestas Lukoševičius <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant