feat(blob): support placeholder fallback for partial updates

[SteNicholas]

Purpose

Linked issue: close #452

A data-evolution partial update rewrites only the touched rows of a blob column and records every untouched row as a placeholder entry (bin_length -2, no data bytes). This PR makes the whole chain understand such entries:

• Blob format: the writer detects an in-band placeholder sentinel value and persists it as a bin_length -2 entry without payload bytes, the same way serialized BlobDescriptors are sniffed by their magic header. The reader fails on placeholder entries by default; the internal format option blob.internal.emit-placeholder-sentinel switches it to emit the 9-byte sentinel (0x01 + "BLOBPLHD") so that the fallback merge can identify placeholders after the batch has passed through schema-mapping readers.
• BlobBunch: keeps all blob files of a bunch, where files sharing a max_sequence_number form one non-overlapping layer. When every file shares a single max_sequence_number, SequentialReadOptimize() keeps the existing concat fast path.
• BlobFallbackBatchReader (new): merges the layers of one blob bunch ordered by max_sequence_number descending; row-id ranges a layer does not cover are padded with placeholder gap segments so all layers step in lockstep; each output row takes the newest non-placeholder layer, a row that is a placeholder in every layer degrades to null, and an explicitly written null wins over older layers. Row-range pushdown is honored per layer, including inside gaps.
• DataEvolutionSplitRead: builds the fallback reader when a blob bunch spans multiple sequence layers, and passes the internal format option through AbstractSplitRead::CreateRawFileReaders.

Tests

• blob_format_writer_test.cpp: golden-bytes layout of placeholder entries (TestWritePlaceholderGoldenBytes), strict vs placeholder-aware read modes including descriptor mode (TestReadPlaceholderStrictAndAwareModes), and selection-bitmap reads over placeholders (TestReadPlaceholderWithSelectionBitmap).
• blob_fallback_batch_reader_test.cpp (new): fallback merge across layers with batch-size and file-batch-size sweeps — basic fallback, leading/middle/trailing gaps, all-placeholder rows degrading to null, null-wins semantics, three layers, misaligned-group and creation-argument failures.
• data_evolution_split_read_test.cpp: BlobBunch keeps all sequence layers, rejects overlaps within one layer, and reports RowCount/SequentialReadOptimize accordingly.
• blob_table_inte_test.cpp (IT): end-to-end partial-update reads — placeholder fallback with null overwrite and _ROW_ID alignment plus a blob_as_descriptor read-mode variant (TestDataEvolutionBlobPartialUpdateFallback), multi-layer layouts with per-row and gap-crossing row-range reads (TestDataEvolutionBlobPartialUpdateMultipleLayers), a compacted four-layer scenario with per-row range reads (TestDataEvolutionBlobPartialUpdateCompactedLayers), and row-range pushdown over updated and untouched rows (TestDataEvolutionBlobPartialUpdateWithRowRanges).

API and Format

• No changes under include/paimon.
• Blob file format: bin_length -2 marks a placeholder entry occupying no file space. Readers without placeholder support already reject such entries with an explicit error, and files without placeholders are unaffected.
• New internal (non user-facing) format option blob.internal.emit-placeholder-sentinel, set only by the data-evolution fallback read path.

Documentation

Internal read/write behavior of data-evolution partial updates; no user-facing documentation change.

Generative AI tooling

Generated-by: Claude Code (Claude Fable 5)

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR adds end-to-end support for data-evolution partial updates on BLOB columns by introducing placeholder entries (bin_length == -2) in the blob format and implementing a multi-layer fallback read path that merges newer/older blob layers row-by-row.

Changes:

• Extend blob write/read format to recognize placeholder entries and (optionally) emit an in-band placeholder sentinel for downstream fallback merging.
• Update DataEvolutionSplitRead::BlobBunch to retain all blob files across max-sequence “layers” and introduce a new BlobFallbackBatchReader to resolve placeholders across those layers.
• Wire an internal per-reader format option through AbstractSplitRead::CreateRawFileReaders and add unit + integration coverage for placeholder fallback, gaps, null-wins behavior, and row-range pushdown.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
test/inte/blob_table_inte_test.cpp	Adds end-to-end integration tests for partial-update placeholder fallback, null-wins semantics, descriptor mode, and row-range pushdown.
src/paimon/format/blob/blob_reader_builder.h	Plumbs internal option to enable placeholder-sentinel emission from the blob reader.
src/paimon/format/blob/blob_format_writer.cpp	Detects in-band placeholder sentinel and writes bin_length = -2 placeholder entries (no payload).
src/paimon/format/blob/blob_format_writer_test.cpp	Adds golden-bytes and strict vs placeholder-aware read-mode tests, including selection-bitmap coverage.
src/paimon/format/blob/blob_file_batch_reader.h	Extends blob reader API to optionally emit placeholder sentinel bytes instead of failing on placeholders.
src/paimon/format/blob/blob_file_batch_reader.cpp	Implements placeholder handling in offsets/contents building and strict-mode failure behavior.
src/paimon/core/operation/data_evolution_split_read.h	Updates BlobBunch to model layered blob files and declares fallback-reader construction.
src/paimon/core/operation/data_evolution_split_read.cpp	Keeps layered blob files, chooses concat vs fallback path, and builds padded per-layer segments for fallback merging.
src/paimon/core/operation/data_evolution_split_read_test.cpp	Updates/expands BlobBunch tests to validate layering rules, row-count behavior, and optimize-path selection.
src/paimon/core/operation/abstract_split_read.h	Adds extra_format_options plumbing to allow per-reader format option overrides (used for placeholder emission).
src/paimon/core/operation/abstract_split_read.cpp	Merges extra_format_options over table options when instantiating file formats/readers.
src/paimon/common/reader/blob_fallback_batch_reader.h	Introduces the fallback batch reader interface and contracts for layered placeholder resolution.
src/paimon/common/reader/blob_fallback_batch_reader.cpp	Implements row-wise fallback merging across sequence layers with gap padding and schema validation.
src/paimon/common/reader/blob_fallback_batch_reader_test.cpp	Adds unit tests covering multi-layer fallback, gaps, all-placeholder => null, null-wins, and validation/misalignment failures.
src/paimon/common/data/blob_defs.h	Defines placeholder bin length, sentinel bytes, internal option key, and sentinel predicate helper.
src/paimon/CMakeLists.txt	Registers the new reader source and unit test in the build.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.