Skip to content

Native Iceberg writes: stop falling back on settings iceberg-java's writer ignores #6955

Open

Description

@andygrove

What is the problem the feature request solves?

Some eligibility rules decline writes over settings that cannot change iceberg-java's output, so the fallback gains nothing:

  • write.parquet.shred-variants=true (requireShredVariantsDisabled). Iceberg 1.11 shreds only top-level variant columns (ParquetFormatModel: shreddingEnabled && variantAnalyzer != null && hasVariantColumns(schema)), and the column type rule already declines a write with a variant column at any depth. Spark 4.x resolves this property into every write, so a session that turns shredding on makes every Iceberg write fall back.
  • An io-impl table property. iceberg-java reads io-impl from catalog properties only (HiveCatalog, HadoopCatalog, RESTSessionCatalog, GlueTableOperations and the other catalogs), never from table properties. requireRecognizedTableFileIO already checks the FileIO the table actually uses.
  • parquet.* keys iceberg-java's writer never reads. For a Spark data write, iceberg-java builds ParquetProperties from Iceberg's own properties (Parquet.WriteBuilder.build). It reads parquet.enable.dictionary from the table and write properties, and from the Hadoop configuration only parquet.columnindex.truncate.length (ParquetWriter) and the codec settings (ParquetCodecFactory). So parquet.block.size, parquet.page.size, parquet.writer.version, parquet.enable.summary-metadata, the reader-side keys, and parquet.enable.dictionary in the Hadoop configuration cannot change a data file, yet requireNoUnvettedParquetMrProperties and requireNoParquetHadoopConfOverrides decline all of them. With S3FileIO, the writer's Hadoop configuration is a fresh new Configuration(), so session spark.hadoop.parquet.* settings never reach it.

Checked against the Iceberg 1.5.2, 1.8.1, 1.10.0 and 1.11.0 sources.

Describe the potential solution

  • Remove requireShredVariantsDisabled, and add a CometIcebergWriteDetectionSuite case showing that a variant column still falls back with shredding on.
  • Remove the io-impl property rule.
  • Replace the two parquet.* deny-alls with an explicit list of the parquet-mr keys iceberg-java's writer does not read, built from parquet-mr's ParquetOutputFormat and ParquetInputFormat constants or a vetted static list, and keep failing closed on keys outside the list. parquet.enable.dictionary (as a table property) and parquet.columnindex.truncate.length stay gated until Native Iceberg writes: translate the Parquet settings that have a parquet-rs equivalent #6954 translates them. Re-check the list against Parquet.java and ParquetWriter.java on every Iceberg version bump.

Additional context

Part of #5643 and the native write epic #5649.

Activity

  1. self-assigned this
    on Nov 10, 2026
  2. comphead commented on Nov 10, 2026

    @comphead
    Contributor

    Same deployment as my comment on #6957: spark.hadoop.parquet.io.stats.enabled=false in the cluster's Spark defaults makes requireNoParquetHadoopConfOverrides decline every native Iceberg write. That rule runs before the S3A rules, so admitting the S3A settings alone would not be enough there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions