Skip to content

feat(delta): prepare InterleaveBits integration - #7597

Draft
ErikBPF wants to merge 2 commits into
apache:mainfrom
ErikBPF:feat/7573-interleave-handoff
Draft

ErikBPF wants to merge 2 commits into
apache:mainfrom
ErikBPF:feat/7573-interleave-handoff

Conversation

@ErikBPF

@ErikBPF ErikBPF commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Refs #7573. This draft does not close the issue.

Rationale for this change

Open a collaboration branch following @dwsmith1983's request. The initial commit supplies an independently runnable Delta reference baseline. @dwsmith1983 plans to contribute his Rust ordering code and establish a working base before handing it back for extension and cleanup.

The accepted issue scope remains Delta-compatible InterleaveBits for one or two INT inputs. Hilbert ordering and broader clustering operations require separate scope; this draft does not claim support for them.

What changes are included in this PR?

  • A manual Scala probe covering exact bytes, null-as-zero, signed boundaries, deterministic integer pairs, mixed literals/columns and controlled constant folding against both Delta interleaving paths.
  • Launch instructions and explicit limitations. No native implementation, expression admission changes or CI integration are included yet.

Based on main at 2f9552d022eb84a79b31b3870c70be9725fdb41f. The initial files do not overlap our six other open upstream PRs, including #7593; this baseline does not depend on that pairing PR.

How are these changes tested?

Apollo: fresh compilation and execution on Spark 4.2.0 / Java 17, Delta 4.4.0 and 4.4.1, stock Spark and released Comet 1.2.0, local[2], 2 GiB driver. Each lane requires 15 named checks; row/schema parity is compared between stock and Comet. An accelerated control verifies Comet is active, while the requested expression can still fall back to Spark.

These are manual reference probes, not reactor tests or native acceptance. Current-main runtime, the future Rust implementation and performance remain unverified. Before readiness, add native plan assertions and CI tests, preserve fallback for unsupported types/arities, and validate the native integration against Delta's implementation and tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant