Skip to content

Add example Variant objects where field_id_size and field_offset_size differ - #125

Open
CurtHagenlocher wants to merge 1 commit into
apache:masterfrom
CurtHagenlocher:variant-asymmetric-object-headers
Open

Add example Variant objects where field_id_size and field_offset_size differ#125
CurtHagenlocher wants to merge 1 commit into
apache:masterfrom
CurtHagenlocher:variant-asymmetric-object-headers

Conversation

@CurtHagenlocher

Copy link
Copy Markdown

Closes #124.

What

Adds two object examples in variant/:

case shape header byte widths
object_wide_offsets 2 fields, >256 bytes of field data 0x06 1-byte field IDs, 2-byte field offsets
object_wide_field_ids 262 distinct field names; small object under nested 0x12 (nested object) 2-byte field IDs, 1-byte field offsets

These are the first examples in which an object's field_id_size and field_offset_size differ. Every existing object example uses 1-byte field IDs and 1-byte field offsets, where the two value_header fields hold the same value, so a reader that reads them from each other's bit positions still decodes correctly. arrow-dotnet shipped exactly that bug and validated cleanly against this corpus the whole time (apache/arrow-dotnet#420).

object_wide_field_ids has 261 fields, which also makes it the first example with is_large set to 1 (a 4-byte num_elements).

Verification

  • regen.py was run end to end and reproduces the four new files byte-identically, along with 49 of the 52 files it generates. See below for the three that differ.
  • Both new examples decode to the expected JSON under an independent spec-literal decoder, which was first cross-checked against the existing object_nested and array_nested examples.
  • Both are rejected by a reader that transposes the two value_header fields: object_wide_offsets fails its end-offset check, object_wide_field_ids runs off the end of the buffer.

Notes for reviewers

Two pre-existing regen.py deviations, unrelated to this change

Noting these since I ran the script; happy to file them separately if useful.

  • primitive_null.metadata and primitive_null.value come out empty, which README "Modification 1" already documents as needing a manual fix-up after every regen.
  • primitive_timestamp.value is timezone dependent. '2025-04-16T12:34:56.78'::Timestamp is interpreted in the Spark session timezone, so the committed file decodes to 2025-04-16 16:34:56Z (generated under UTC-4) while a run on a US Pacific machine produces 19:34:56Z. Anyone regenerating the corpus will silently rewrite this file.

object_wide_offsets and object_wide_field_ids are the first examples in which an
object's field_id_size and field_offset_size differ. Every existing object
example uses 1-byte field IDs and 1-byte field offsets, where the two
value_header fields hold the same value, so a reader that reads them from each
other's bit positions still decodes correctly and the corpus cannot detect that
class of bug.

- object_wide_offsets: two fields (1-byte field IDs) and more than 256 bytes of
  field data (2-byte field offsets), giving header byte 0x06.
- object_wide_field_ids: 262 distinct field names force 2-byte field IDs, while
  the object under the "nested" field holds a single small value so its own
  field offsets still fit in 1 byte, giving header byte 0x12. Its enclosing
  object has 261 fields, which also makes this the first example with is_large
  set to 1.

Verified by running regen.py end to end: it reproduces the four new files
byte-identically, along with 49 of the 52 files it generates. The three that
differ are pre-existing and unrelated -- primitive_null.metadata and
primitive_null.value, which Spark emits empty and which README "Modification 1"
documents as hand-corrected, and primitive_timestamp.value, which encodes the
generating machine's session timezone.

The new examples were also checked to decode to the expected JSON, and to be
rejected by a reader that transposes the two value_header fields.

object_wide_field_ids also covers the first case listed in apache#78 (an object with
more than 2**8 fields, requiring a 4-byte num_elements). The remaining cases
there -- more than 2**16 and 2**24 distinct fields -- are still open, and their
TODOs in regen.py now sit under a reference to apache#78 rather than apache#77. They are
also reworded, since the number of distinct fields drives the field ID width
rather than the field offset width.

Closes apache#124.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add example Variant objects where field_id_size and field_offset_size differ

1 participant