Native Iceberg writes: admit S3 settings that cannot change the written object, and translate S3 credential providers and encryption settings #6957

What is the problem the feature request solves?

For an s3 or s3a data location, the eligibility gate falls back on every explicit Hadoop fs.s3a.* setting other than the access key, secret key, session token, endpoint, endpoint region and path-style access (it also ignores a few read and committer settings), and on every Iceberg FileIO s3.* or client.* property outside its allowlist (#6139, PR #6441). Clusters commonly set such keys cluster-wide, so a HadoopFileIO table on S3 rarely passes the gate in production. Each of these makes a write fall back today:

• fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem, which many Spark configurations set explicitly
• connection and thread pools, timeouts and retries: fs.s3a.connection.maximum, fs.s3a.threads.max, fs.s3a.connection.timeout, fs.s3a.attempts.maximum, fs.s3a.retry.limit
• upload buffering: fs.s3a.fast.upload.buffer, fs.s3a.multipart.size, fs.s3a.multipart.threshold, fs.s3a.buffer.dir, fs.s3a.block.size
• fs.s3a.aws.credentials.provider
• fs.s3a.encryption.algorithm and fs.s3a.encryption.key, and the older fs.s3a.server-side-encryption-algorithm and fs.s3a.server-side-encryption.key
• on S3FileIO: s3.multipart.part-size-bytes, s3.multipart.num-threads, s3.multipart.threshold, s3.staging-dir, s3.retry.*, s3.delete-enabled, s3.delete.batch-size, s3.preload-client-enabled, client.credentials-provider, s3.acl, s3.write.storage-class

Describe the potential solution

Three parts, each keeping the gate an allowlist:

1. Settings that cannot change the written object, where it is written, or who writes it. Admit them, as IgnoredHadoopS3Keys already does for the committer and read settings: the native writer has its own connection pool, retries and upload buffering. Audit the candidates against Hadoop's org.apache.hadoop.fs.s3a.Constants and Iceberg's S3FileIOProperties and AwsClientProperties for each pinned version. Admit fs.s3a.impl only when it names S3AFileSystem. Keep failing closed for anything that can change the request: endpoint and region selection, SSL, proxies, signing, requester pays, ACLs, storage class, checksums, access points.
2. Credentials. Map the well-known provider classes named in fs.s3a.aws.credentials.provider and client.credentials-provider to native credential sources, as the native Parquet scan already does in native/core/src/parquet/objectstore/s3.rs (instance profile, container, environment, web identity, profile, simple, temporary and anonymous). An unrecognized provider class still falls back.
3. Server-side encryption, ACL, storage class, requester pays and dsse-kms. OpenDAL's S3 backend supports all of these (server_side_encryption*, default_acl, default_storage_class, enable_request_payer), but iceberg-rust's S3 property parser maps only the s3.sse.type values none, s3, kms and custom. Translate the S3A encryption settings to s3.sse.*, and for the rest either extend iceberg-rust's parser upstream or set the options in Comet's own S3 storage factory.

Each admitted or translated setting needs a CometIcebergWriteDetectionSuite case, and the translated ones a write through both writers against MinIO.

Additional context

Part of #5643 and the native write epic #5649. S3 object tags, access points, transfer acceleration, dual-stack and cross-region endpoints, remote signing and custom S3A signers stay a fallback: Comet's native S3 client has no equivalent for them.