[core] Introduce portable TableDescriptor for cross-language table construction

[JunRuiLee]

Purpose

Introduce a portable, versioned JSON description of a table — TableDescriptor — so a non-Java runtime (e.g. a paimon-rust / paimon-cpp reader embedded in a compute engine) can rebuild a table for scan/read without a catalog lookup.

Today a Java engine can serialize a Table via native Java serialization, but that blob is not consumable by non-Java readers (it carries JVM-only state such as FileIO / CatalogEnvironment). As a result, an embedding engine has to re-resolve the table through a catalog on the reader side, which can observe a different schema/snapshot than the one the planner used, and ties the reader to a derived filesystem-catalog layout.

This PR adds a slim, language-neutral contract that captures only what a reader needs:

• org.apache.paimon.table.TableDescriptor (paimon-api, @Experimental): version, path, the full TableSchema (identical to an on-disk schema/schema-N), optional options / database / name / branch. Serialized through JsonSerdeUtil so the nested TableSchema uses Paimon's custom serde; unknown fields are ignored for forward compatibility; null fields are omitted to keep the payload slim.
• org.apache.paimon.table.TableDescriptorSerializer (paimon-core, @Experimental): builds a descriptor from a resolved FileStoreTable (location() + schema()), and serializes it to JSON. A non-main branch is carried verbatim so a reader can fail fast rather than silently read main; database/name must be set together or not at all.

The reader side (in the paimon-rust cross-language reader) consumes the same JSON and reconstructs the table directly; a byte-identical golden fixture (table-descriptor-v1.json) pins the wire form on both sides.

Tests

• TableDescriptorTest (paimon-api): JSON round-trip; nested TableSchema uses Paimon's custom serde; unknown fields ignored.
• TableDescriptorSerializerTest (paimon-core): build from a FileStoreTable; round-trip; partial-identifier rejection; non-main branch carried through.
• TableDescriptorGoldenTest (paimon-core): producer output equals the checked-in canonical golden, and the golden parses back — the exact wire form the cross-language reader is tested against.

API and Format

• New @Experimental public API: TableDescriptor (paimon-api) and TableDescriptorSerializer (paimon-core).
• New JSON descriptor format, versioned (version = 1). No change to any existing on-disk or wire format.

Documentation

None yet (experimental API). Docs can follow once the contract stabilizes.

---

Motivated by the Apache Doris Paimon-external-table integration (apache/doris#65883), where the compute node currently re-resolves the table through a catalog on the backend; this contract lets it consume the FE-planned table instead.

[JingsongLi]

1. [P2] Composite tables are silently downgraded to regular tables

   • TableDescriptorSerializer.java:67 only records location + schema.
   • The location() and schema() methods of FallbackReadFileStoreTable and ChainGroupReadTable are directly delegated to the primary table, so the descriptors generated for composite tables are exactly the same as those for regular primary tables.
   • After reconstruction on the Rust side, the partition-level merge/fallback semantics for dual branches are lost, which may result in silently skipping or incorrectly reading data.
   • I wrote a temporary test to confirm that the JSON of fallback tables and regular tables is byte-for-byte identical.
   • It is recommended that v1 at least explicitly reject these wrappers; tableKind and branch/composition information can be added in future versions.

2. [P2] Dynamic configurations and credentials are included in plain text within the descriptor

   • AbstractFileStoreTable.copyInternal merges all dynamic options into tableSchema.options, which is then exported in its entirety by TableDescriptorSerializer.java:67.
   • Testing confirmed that the supported fs.s3a.secret.key=review-secret appears verbatim in the JSON.
   • The Rust FFI already accepts storage_options separately, so the descriptor should no longer carry AK/SK.
   • It is recommended to use a persistent schema and include necessary dynamic read options via an explicit allowlist; filter out storage, catalog, and sensitive configurations.

There is also a contract issue: the top-level options is a public field, but the Java producer always sets it to null, and Rust v1 explicitly ignores it; it is best to remove it or implement fail-fast behavior for non-null values to prevent users from mistakenly believing it will take effect.

[JunRuiLee]

1. [P2] Composite tables are silently downgraded to regular tables

   • TableDescriptorSerializer.java:67 only records location + schema.
   • The location() and schema() methods of FallbackReadFileStoreTable and ChainGroupReadTable are directly delegated to the primary table, so the descriptors generated for composite tables are exactly the same as those for regular primary tables.
   • After reconstruction on the Rust side, the partition-level merge/fallback semantics for dual branches are lost, which may result in silently skipping or incorrectly reading data.
   • I wrote a temporary test to confirm that the JSON of fallback tables and regular tables is byte-for-byte identical.
   • It is recommended that v1 at least explicitly reject these wrappers; tableKind and branch/composition information can be added in future versions.

2. [P2] Dynamic configurations and credentials are included in plain text within the descriptor

   • AbstractFileStoreTable.copyInternal merges all dynamic options into tableSchema.options, which is then exported in its entirety by TableDescriptorSerializer.java:67.
   • Testing confirmed that the supported fs.s3a.secret.key=review-secret appears verbatim in the JSON.
   • The Rust FFI already accepts storage_options separately, so the descriptor should no longer carry AK/SK.
   • It is recommended to use a persistent schema and include necessary dynamic read options via an explicit allowlist; filter out storage, catalog, and sensitive configurations.

There is also a contract issue: the top-level options is a public field, but the Java producer always sets it to null, and Rust v1 explicitly ignores it; it is best to remove it or implement fail-fast behavior for non-null values to prevent users from mistakenly believing it will take effect.

Thanks for the careful review, @JingsongLi! All three addressed:

1. Composite tables silently downgraded — Fixed. TableDescriptorSerializer.from now rejects DelegatedFileStoreTable (covers fallback / chain / privileged) with an UnsupportedOperationException, so a composite table can no longer be serialized as if it were a plain one.

2. Dynamic options / credentials leaked in plaintext — Fixed. The serializer now ships the persisted schema (table.schemaManager().schema(table.schema().id())) instead of table.schema(), so the dynamic options merged in by copyInternal (incl. fs.s3a.secret.key and the injected path) no longer appear in the descriptor. Storage credentials are passed to the reader out of band. Added a test asserting fs.s3a.secret.key does not appear in the JSON.

3. Misleading top-level options — Removed the field from the DTO entirely (it was always null and ignored by the reader).

One addition on top of the review: instead of rejecting non-main branches, v1 now supports a specific branch of a plain table — the branch travels in a dedicated branch field (omitted for main), and since schemaManager() is branch-scoped the persisted schema is read from the branch. Composite/fallback/chain tables remain out of scope.

The reader side (paimon-rust) will be updated to consume this in its own PR once this contract lands.

[JingsongLi]

• TableDescriptorSerializer.java:80 has been modified to read the persisted schema-N, which effectively prevents the leakage of dynamic credentials and correctly rejects delegated/composite tables and non-main branches.
• However, the input table may be a copy containing scan.snapshot-id, scan.tag-name, or scan.timestamp[-millis]; these dynamic options are removed by the persisted schema, and the descriptor lacks snapshot/time-travel fields.
• The Rust reader reconstructs regular tables based solely on the descriptor and ultimately reads the latest snapshot rather than the one seen by the planner, which constitutes silent reading of incorrect data.

[JunRuiLee]

Thanks @JingsongLi. Now the serializer resolves the snapshot at serialize time (TimeTravelUtil.tryTravelToSnapshot) and carries only the resolved snapshotId (null for ordinary scans) — carrying the resolved id rather than raw scan.* avoids duplicating TimeTravelUtil.SCAN_KEYS / selector drift and keeps storage/sensitive options out. The reader (paimon-rust) will pin snapshotId in its own PR; on the split-read path the split already pins files.

[JingsongLi]

I'm still a bit confused—why are we introducing this? Doesn't TableSchema alone suffice? The rest of the information seems to be in “Identifier” or “dynamic options,” all of which are related to the calculation engine.

[JunRuiLee]

I'm still a bit confused—why are we introducing this? Doesn't TableSchema alone suffice? The rest of the information seems to be in “Identifier” or “dynamic options,” all of which are related to the calculation engine.

Thanks, I think this concern is valid. I’ll discuss this direction with the Apache Doris community and follow up once we’ve aligned on the design.