[client][rust] Align fixed-schema scans by column ID#3771
Conversation
|
While reviewing the Rust Arrow fixed-schema read path, I noticed that historical data is currently aligned by field name rather than by stable column ID. Under typical schema-evolution semantics, this could prevent old data from being matched correctly after a column rename, so I initially considered it a bug. However, after local testing and further investigation, I confirmed that Fluss currently supports only a limited set of schema-evolution operations, such as adding nullable columns. Although the client exposes a Nevertheless, the Arrow fixed-schema path is inconsistent with the KV |
Purpose
Linked issue: close #3770
The Rust fixed-schema log scanner previously aligned historical Arrow batches with the scanner schema by field name.
This behavior was inconsistent with the KV
FixedSchemaDecoder, which uses stable Fluss column IDs. Name-based alignment may lose values when a column keeps the same ID but changes its name, or incorrectly reuse values whendifferent column IDs have the same name.
This PR aligns fixed-schema log reads by column ID, consistent with
FixedSchemaDecoder.Brief change log
index_mappingutility, using stable column IDs.ReadContextfor both local and remote log reads.Tests
Added Rust unit tests.
API and Format
No public API or storage format changes.
Documentation
No documentation changes are required.