Skip to content

[client][rust] Align fixed-schema scans by column ID#3771

Open
slfan1989 wants to merge 1 commit into
apache:mainfrom
slfan1989:fluss-3770
Open

[client][rust] Align fixed-schema scans by column ID#3771
slfan1989 wants to merge 1 commit into
apache:mainfrom
slfan1989:fluss-3770

Conversation

@slfan1989

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #3770

The Rust fixed-schema log scanner previously aligned historical Arrow batches with the scanner schema by field name.

This behavior was inconsistent with the KV FixedSchemaDecoder, which uses stable Fluss column IDs. Name-based alignment may lose values when a column keeps the same ID but changes its name, or incorrectly reuse values when
different column IDs have the same name.

This PR aligns fixed-schema log reads by column ID, consistent with FixedSchemaDecoder.

Brief change log

  • Capture the fixed target Fluss schema when creating a fixed-schema log scanner.
  • Build source-to-target mappings with the existing index_mapping utility, using stable column IDs.
  • Pass the mapping to Arrow ReadContext for both local and remote log reads.
  • Preserve values when source and target columns have the same ID.
  • Fill target columns with NULL when their IDs do not exist in the source schema.
  • Prevent columns with the same name but different IDs from being incorrectly matched.
  • Add regression tests for local and remote fixed-schema decoding.

Tests

Added Rust unit tests.

API and Format

No public API or storage format changes.

Documentation

No documentation changes are required.

@slfan1989

Copy link
Copy Markdown
Contributor Author

While reviewing the Rust Arrow fixed-schema read path, I noticed that historical data is currently aligned by field name rather than by stable column ID. Under typical schema-evolution semantics, this could prevent old data from being matched correctly after a column rename, so I initially considered it a bug.

However, after local testing and further investigation, I confirmed that Fluss currently supports only a limited set of schema-evolution operations, such as adding nullable columns. Although the client exposes a RENAME COLUMN interface, the corresponding functionality has not yet been implemented on the server. Therefore, this issue cannot currently be reproduced through the complete client-server workflow.

Nevertheless, the Arrow fixed-schema path is inconsistent with the KV FixedSchemaDecoder in how fields are aligned. To provide consistent semantics across both read paths and establish a reliable foundation for supporting additional schema-evolution operations in the future, I submit a PR that changes Arrow fixed-schema alignment from field names to stable column IDs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement][rust] Align fixed-schema log reads by column ID

1 participant