Skip to content

Align PutObject checksum error messages with S3 #226

Description

@cbornet

Expected Behavior

  • Should return The CRC32 you specified did not match the calculated checksum. when a PutObject fails due to a CRC32 checksum mismatch.
  • Should return The CRC32C you specified did not match the calculated checksum. when a PutObject fails due to a CRC32C checksum mismatch.

Current Behavior

  • Returns The provided 'x-amz-checksum' header does not match what was computed. when a PutObject fails due to a checksum mismatch.

Possible Solution

Steps to Reproduce (for bugs)

  1. Launch a local silo server
  2. Run the script
import boto3
import zlib
import base64
import tempfile
import os
from botocore.exceptions import ClientError

os.environ["AWS_REQUEST_CHECKSUM_CALCULATION"] = "when_required"
CHUNK_SIZE = 1 * 1024 * 1024  # 8 MB

AWS_REGION = "us-east-1"
BUCKET_NAME = "your-unique-bucket-name"


def crc32_to_base64(crc):
    crc &= 0xFFFFFFFF
    return base64.b64encode(crc.to_bytes(4, "big")).decode("utf-8")


def create_s3_client():
    return boto3.client(
        "s3",
        endpoint_url="http://localhost:9000",
        aws_access_key_id="minioadmin",
        aws_secret_access_key="minioadmin",
        region_name=AWS_REGION
    )


def create_bucket_if_not_exists(s3, bucket_name):
    try:
        s3.head_bucket(Bucket=bucket_name)
        print(f"Bucket already exists: {bucket_name}")
    except ClientError:
        print(f"Creating bucket: {bucket_name}")

        if AWS_REGION == "us-east-1":
            s3.create_bucket(Bucket=bucket_name)
        else:
            s3.create_bucket(
                Bucket=bucket_name,
                CreateBucketConfiguration={
                    "LocationConstraint": AWS_REGION
                }
            )


def create_temp_file():
    tmp = tempfile.NamedTemporaryFile(delete=False)
    target_size = CHUNK_SIZE * 1  # 3 parts
    chunk = b"A" * (1024 * 1024)  # write in 1 MB blocks

    written = 0
    while written < target_size:
        to_write = min(len(chunk), target_size - written)
        tmp.write(chunk[:to_write])
        written += to_write
    tmp.close()
    return tmp.name


def upload(s3, file_path, bucket, key):
    with open(file_path, "rb") as f:
        data = f.read()

    crc32 = zlib.crc32(data) + 1
    crc32_b64 = crc32_to_base64(crc32)

    result = s3.put_object(
        Bucket=bucket,
        Key=key,
        Body=data,
        ChecksumCRC32=crc32_b64,
    )

    print("Upload complete ✅")
    print("S3 checksum:", result.get("ChecksumCRC32"))


if __name__ == "__main__":
    s3 = create_s3_client()

    # 1. Ensure bucket exists
    create_bucket_if_not_exists(s3, BUCKET_NAME)

    # 2. Create temp file
    temp_file = create_temp_file()

    try:
        # 3. Upload
        upload(
            s3,
            file_path=temp_file,
            bucket=BUCKET_NAME,
            key="uploads/temp-example.txt"
        )
    finally:
        os.remove(temp_file)
        print("Temp file deleted")

Context

Aligning Silo with S3

Regression

No

Your Environment

  • Version used (silo --version): RELEASE.2026-09-16T00-00-00Z
  • Server setup and configuration:
  • Operating System and version (uname -a):

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    • Status
      In progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions