Repository navigation
feat: add a native kernel for Delta's InterleaveBits - #7600
Draft
dwsmith1983 wants to merge 1 commit into
Draft
dwsmith1983 wants to merge 1 commit into
dwsmith1983 wants to merge 1 commit into
Conversation
Adds delta_interleave_bits, a scalar function that builds Delta Lake's Z-order key for one or two Int32 inputs: bits are taken from the most significant down, alternating between inputs with the first input in the higher position, a null input counts as zero, and the result is four bytes per input and never null. The tests check the output against values produced by Delta's own InterleaveBits on both its default and fast code paths, including signed extrema and nulls. Nothing calls the function from Spark yet; the Delta serde that uses it comes separately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Refs #7573. This is the first of three PRs:
1 and 2 can merge in either order; #7597 goes last.
Rationale for this change
Delta's
InterleaveBitsbuilds the Z-order key for liquid clustering andOPTIMIZE ZORDER BY. Comet has no native version, so the projection falls back to Spark. This adds the kernel on its own, so it can be reviewed without the Delta wiring.What changes are included in this PR?
delta_interleave_bits, a scalar function for one or twoInt32inputs. Bits are taken from the most significant down, alternating between inputs with the first input in the higher position. A null input counts as zero, and the result is four bytes per input and never null, as in Delta.all_scalar_functions(). Nothing calls it from Spark yet.How are these changes tested?
Unit tests compare the output with values produced by Delta's own
InterleaveBitson both its default and fast code paths: signed extrema, nulls, alternating bit patterns and generated pairs. Other tests cover scalar and array arguments in each position, all-scalar calls, and rejection of wrong argument counts or types.