Skip to content

feat: Add configurable S3 metadata headers to catalog - #221

Open
xieandrew wants to merge 3 commits into
masterfrom
220-s3-meta-headers
Open

xieandrew wants to merge 3 commits into
masterfrom
220-s3-meta-headers

Conversation

@xieandrew

Copy link
Copy Markdown
Collaborator

Closes #220.

Example configuration:

.ice-rest-catalog.yaml

uri: jdbc:sqlite:file:data/ice-rest-catalog/db.sqlite?journal_mode=WAL&synchronous=OFF&journal_size_limit=500
warehouse: s3://bucket1
s3:
  endpoint: http://localhost:8999
  pathStyleAccess: true
  accessKeyID: miniouser
  secretAccessKey: miniopassword
  metadata:
    x-amz-meta-expiration-seconds: "86400"
bearerTokens:
  - value: foo

@xieandrew xieandrew added the ice-rest-catalog Relates to ice-rest-catalog label Sep 21, 2026
@xieandrew
xieandrew requested a review from subkanthi September 21, 2026 15:52
@subkanthi

Copy link
Copy Markdown
Collaborator

LLM Review, please check if its valid

The CRT async client path does not install the `S3ObjectMetadataInterceptor`, unlike the standard sync and async paths above. When `s3.crt.enabled` is enabled, `PutObjectRequest` and `CreateMultipartUploadRequest` will therefore omit the configured `s3.metadata.*` headers, so this feature silently stops working for that supported client mode.

Please apply `applyObjectMetadataConfiguration` to the CRT builder as well, and add coverage for the CRT path if feasible.

# Conflicts:
#	ice-rest-catalog/src/test/pyiceberg/uv.lock
@xieandrew

Copy link
Copy Markdown
Collaborator Author

Review finding is not valid:

  • The async client never creates objects. The async client is passed only to toInputFile() / S3InputFile which reads. So no PutObjectRequest or CreateMultipartUploadRequest is ever issued by the async client, CRT or otherwise.
  • The CRT builder cannot take an interceptor. There is no supported way to install an ExecutionInterceptor on a CRT client.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ice-rest-catalog Relates to ice-rest-catalog

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support configurable x-amz-meta-* headers for S3 requests

2 participants