Add example Variant objects where field_id_size and field_offset_size differ - #125
Open
CurtHagenlocher wants to merge 1 commit into
Open
Conversation
object_wide_offsets and object_wide_field_ids are the first examples in which an object's field_id_size and field_offset_size differ. Every existing object example uses 1-byte field IDs and 1-byte field offsets, where the two value_header fields hold the same value, so a reader that reads them from each other's bit positions still decodes correctly and the corpus cannot detect that class of bug. - object_wide_offsets: two fields (1-byte field IDs) and more than 256 bytes of field data (2-byte field offsets), giving header byte 0x06. - object_wide_field_ids: 262 distinct field names force 2-byte field IDs, while the object under the "nested" field holds a single small value so its own field offsets still fit in 1 byte, giving header byte 0x12. Its enclosing object has 261 fields, which also makes this the first example with is_large set to 1. Verified by running regen.py end to end: it reproduces the four new files byte-identically, along with 49 of the 52 files it generates. The three that differ are pre-existing and unrelated -- primitive_null.metadata and primitive_null.value, which Spark emits empty and which README "Modification 1" documents as hand-corrected, and primitive_timestamp.value, which encodes the generating machine's session timezone. The new examples were also checked to decode to the expected JSON, and to be rejected by a reader that transposes the two value_header fields. object_wide_field_ids also covers the first case listed in apache#78 (an object with more than 2**8 fields, requiring a 4-byte num_elements). The remaining cases there -- more than 2**16 and 2**24 distinct fields -- are still open, and their TODOs in regen.py now sit under a reference to apache#78 rather than apache#77. They are also reworded, since the number of distinct fields drives the field ID width rather than the field offset width. Closes apache#124. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #124.
What
Adds two object examples in
variant/:object_wide_offsets0x06object_wide_field_idsnested0x12(nested object)These are the first examples in which an object's
field_id_sizeandfield_offset_sizediffer. Every existing object example uses 1-byte field IDs and 1-byte field offsets, where the twovalue_headerfields hold the same value, so a reader that reads them from each other's bit positions still decodes correctly. arrow-dotnet shipped exactly that bug and validated cleanly against this corpus the whole time (apache/arrow-dotnet#420).object_wide_field_idshas 261 fields, which also makes it the first example withis_largeset to 1 (a 4-bytenum_elements).Verification
regen.pywas run end to end and reproduces the four new files byte-identically, along with 49 of the 52 files it generates. See below for the three that differ.object_nestedandarray_nestedexamples.value_headerfields:object_wide_offsetsfails its end-offset check,object_wide_field_idsruns off the end of the buffer.Notes for reviewers
data_dictionary.jsonis edited in place rather than rewritten, since the committed file is hand-maintained (it carries a trailing comma, andprimitive_timestampntzalready precedesprimitive_timestamp_nanos).object_wide_field_idsalso covers the first case listed in Add example Variant values with larger numbers of fields / array elements #78. The 2**16 and 2**24 cases there remain open; their TODOs inregen.pynow sit under a reference to Add example Variant values with larger numbers of fields / array elements #78 rather than Add example nested Variant values that are not JSON encodeable #77, and are reworded, since the number of distinct fields drives the field ID width rather than the field offset width.Two pre-existing regen.py deviations, unrelated to this change
Noting these since I ran the script; happy to file them separately if useful.
primitive_null.metadataandprimitive_null.valuecome out empty, which README "Modification 1" already documents as needing a manual fix-up after every regen.primitive_timestamp.valueis timezone dependent.'2025-04-16T12:34:56.78'::Timestampis interpreted in the Spark session timezone, so the committed file decodes to2025-04-16 16:34:56Z(generated under UTC-4) while a run on a US Pacific machine produces19:34:56Z. Anyone regenerating the corpus will silently rewrite this file.