docs: lead README with the Arrow-native framing by andygrove · Pull Request #4428 · apache/datafusion-comet

andygrove · 2026-05-25T17:11:33Z

Which issue does this PR close?

Part of #4419 (the documentation-only phase, first slice).

Rationale for this change

Comet's documentation conflates several distinct ideas under the word "native": implementation language (Rust vs JVM), pipeline membership (handled by Comet vs falls back to Spark), and data format (Arrow columnar vs Spark rows). Issue #4419 spells out a clearer vocabulary so the docs stop overloading "native" and can scale to a roadmap where some JVM code paths (today Scala UDF codegen; soon Arrow UDFs and hybrid impls) also live inside the Comet pipeline.

The original version of this PR rolled the new vocabulary into ~20 files at once. This revision narrows the scope to a single file — the top-level README.md — so reviewers can sign off on the framing first. The user-guide and contributor-guide sweeps will follow as separate PRs.

What changes are included in this PR?

README.md only. The two top paragraphs and the "What Comet Accelerates" list are rewritten:

The opening paragraph leads with the Arrow-native pipeline (operators, expressions, shuffle, and broadcast all stay in Apache Arrow columnar format) rather than "native Rust implementations on Apache DataFusion."
The intro to "What Comet Accelerates" makes the Rust-vs-JVM split explicit: most operators and expressions run as Rust on DataFusion; some run as JVM code over Arrow batches. Either way the work stays in the Comet pipeline.
The shuffle bullet upgrades from "native columnar shuffle with support for hash and range partitioning" to "Arrow-IPC columnar shuffle ... in a native Rust implementation paired with a JVM fallback for unsupported partition key types."
A new bullet introduces the experimental Scala/Java UDF support and links to the existing user-guide page.

No other docs files are touched. Operator renames (CometExchange → CometNativeShuffleExchange, etc.) and the user-guide / contributor-guide vocabulary sweep are explicitly out of scope here and will land in follow-on PRs.

How are these changes tested?

Documentation only. Verified that the README renders correctly on GitHub and that the new wording matches the vocabulary rules in #4419.

Rewrite the top two paragraphs of README.md so the value prop leads with the Arrow-native pipeline (operators, expressions, shuffle, and broadcast all in Apache Arrow columnar format) rather than 'native Rust implementations'. The accelerator list grows by one entry to mention the experimental Scala/Java UDF support; shuffle and 'What Comet Accelerates' wording is tightened to match. No other docs are touched in this PR. Contributor-guide and user-guide prose updates for the same vocabulary clean-up (apache#4419) will follow separately.

mbutrovich · 2026-05-29T19:47:29Z

 - **Apache Iceberg**: accelerated Parquet scans when reading Iceberg tables from Spark
  (see the [Iceberg guide](https://datafusion.apache.org/comet/user-guide/iceberg.html))
- **Shuffle**: native columnar shuffle with support for hash and range partitioning
+- **Shuffle**: Arrow-IPC columnar shuffle with support for hash and range partitioning, in a native Rust


Didn't we add a (not 100% Spark-compatible) round-robin partitioning solution? Should we skip that if it's opt-in?

Yes, round-robin in native is not compatible with Spark's version, so it is opt-in. For the top level README it seems like too much information to try and include

mbutrovich · 2026-05-29T19:48:06Z

  map, JSON, hash, and predicate categories
 - **Aggregations**: hash aggregate with support for `FILTER (WHERE ...)` clauses
 - **Joins**: hash join, sort-merge join, and broadcast join
+- **Scala/Java UDFs**: experimental support for keeping Scala/Java scalar UDFs in the Comet pipeline


We can drop "experimental" if #4514 merges first.

Co-authored-by: Matt Butrovich <[email protected]>

apache#4514 (JVM Scala UDF codegen dispatch enabled by default) has merged, so the support is no longer experimental.

comphead · 2026-06-10T20:51:51Z

  map, JSON, hash, and predicate categories
 - **Aggregations**: hash aggregate with support for `FILTER (WHERE ...)` clauses
 - **Joins**: hash join, sort-merge join, and broadcast join
+- **Scala/Java UDFs**: support for keeping Scala/Java scalar UDFs in the Comet pipeline


what about Python UDFs?

Python UDF support did not merge yet

comphead · 2026-06-10T20:52:23Z

+- **Scala/Java UDFs**: support for keeping Scala/Java scalar UDFs in the Comet pipeline
+  via Spark's whole-stage codegen (see the
+  [Scala UDF guide](https://datafusion.apache.org/comet/user-guide/scala_java_udfs.html))



we can also add windows

Co-authored-by: Oleks V <[email protected]>

comphead

Thanks @andygrove the PR looks solid IMO

Use the apache#4419 terminology on docs/source/index.md, which the README rewrite in apache#4428 did not cover. The page now says what Comet does with DataFusion rather than calling it a native engine. Part of apache#4419. Generated-by: OpenCode

Use the apache#4419 terminology on docs/source/index.md, which the README rewrite in apache#4428 did not cover. The page now says what Comet does with DataFusion rather than calling it a native engine. Also drop "powerful" from the edited paragraph, and link the benchmarking guide by its relative path rather than its published URL. Part of apache#4419. Generated-by: OpenCode

Use the apache#4419 terminology on docs/source/index.md, which the README rewrite in apache#4428 did not cover. The page now says what Comet does with DataFusion rather than calling it a native engine, and the opening line carries the README's Arrow framing, as apache#4419 asks. Also drop "powerful" from the edited paragraph, and link the benchmarking guide by its relative path rather than its published URL. Part of apache#4419. Generated-by: OpenCode

Use the apache#4419 terminology on docs/source/index.md, which the README rewrite in apache#4428 did not cover. The page now says what Comet does with DataFusion rather than calling it a native engine, and the opening line carries the README's Arrow framing, as apache#4419 asks. Say that Comet hands DataFusion the operators it runs in Rust, not all supported operators, since some run on the JVM. Also drop "powerful" from the edited paragraph, and link the benchmarking guide by its relative path rather than its published URL. Part of apache#4419. Generated-by: OpenCode

…ages (apache#7151) * docs: add a documentation style guide Add docs/source/contributor-guide/style_guide.md, which writes down the vocabulary from apache#4419 (pipeline membership, implementation language and data format), the rule for "native", how to name operators in prose, and the page conventions that the docs-review skill in apache#4420 cites. Each rule has a stable ID so that reviews can cite it. The guide follows the code where apache#4419 and the code differ: "Hybrid" keeps the meaning it already has in expressions.md, expressions are described separately from plan nodes, and "Arrow-native" is a claim about data format only. The Generated Content section names the generated text exactly, by the markers GenerateDocs fills in, including the config tables built from QueryPlanSerde and contrib config classes, and the cast tables built from CometCast. P18 covers attribution of third-party content, which apache#4420 lists among its checks. The guide's scope covers .doc strings in contrib config classes, and its vocabulary and accuracy rules also apply to doc comments, AGENTS.md and agent skills. Link the page from the contributor guide toctree and from AGENTS.md. Part of apache#4419. Generated-by: OpenCode * docs: correct the plugin overview and adopt the apache#4419 terminology Correct statements in plugin_overview.md that no longer match the code: - CometSparkSessionExtensions registers more than CometRule: it also registers rules that run after columnar transitions are inserted, AQE query stage optimizer rules on Spark 3.5 and later, and IcebergWriteStrategy. - Comet does replace part of a query stage, and an unsupported operator stays in Spark, not the whole plan. Describe the post-transition rules in the order CometRule runs them: RevertNativeForTransitionHeavyStages first, which can decline an unsafe revert, then EliminateRedundantTransitions, which uses CometColumnarToRowExec, or CometNativeColumnarToRowExec when that opt-in config is set and the converter supports the schema. - Not every leaf of the Rust plan is a ScanExec. ScanExec takes Arrow batches from the JVM; a direct shuffle read of a Comet shuffle with supported types is a ShuffleScanExec; the Parquet, Iceberg and CSV scans read their files in Rust. - Blocks are formed from consecutive CometNativeExec operators, except that the write operators start a new block for their child. The native shuffle writer absorbs a CometNativeExec child. Add a paragraph on the JVM codegen dispatcher, which apache#4419 asks this page to cover: some expressions in a Rust plan run Spark's generated JVM code over Arrow batches through JNI, and the operator around them stays in the Comet pipeline. Scala and Java UDFs take the same path. Also replace the vague uses of "native" with the apache#4419 terms. The scan section (Phase 1) is left alone, because apache#6052 rewrites it. Part of apache#4419. Generated-by: OpenCode * docs: adopt the apache#4419 terminology in the native shuffle guide Replace vague uses of "native" in native_shuffle.md. Native shuffle accepts any CometPlan child, so call the child a Comet operator rather than a native one. Where "native" was used as a noun, name the Rust component instead. Also correct statements this touched: - When RevertNativeForTransitionHeavyStages restores a stage, a native shuffle whose input came from spark.comet.convert.shuffleInput.enabled switches back to JVM columnar shuffle. - When direct read cannot serve a shuffle stage, the stage stays as it is and the operators that read it cannot join a Rust plan. The check is QueryPlanSerde.supportedDataType; supportedSinkDataType does not exist. - Drop the unverified reason given for rejecting nested range keys. - Describe spark.comet.shuffle.directRead.enabled the same way as the JVM shuffle guide. Contributor-guide forms such as "native code" and "native plan", the "native shuffle" name, and config keys are unchanged. Part of apache#4419. Generated-by: OpenCode * docs: adopt the apache#4419 terminology in the JVM shuffle guide Replace vague uses of "native" in jvm_shuffle.md. Say which parts of JVM shuffle run on the JVM (partitioning) and which run in Rust (Arrow IPC encoding), and call a shuffle's child a Comet operator, since native shuffle accepts any CometPlan child, including JVM-side ones. Say that direct read applies when a Comet block reads the shuffle as a ShuffleScan input. Also describe spark.comet.shuffle.directRead.enabled accurately: both read paths decode in Rust, and direct read decodes inside the Rust plan, bypassing Arrow FFI, as the config's own doc string says. Part of apache#4419. Generated-by: OpenCode * docs: adopt the apache#4419 terminology in the Gluten comparison Replace vague uses of "native" with the term each sentence means: the Comet pipeline, Rust, or Velox on the Gluten side. Describe Velox and DataFusion as columnar query engines rather than vectorized ones. Two factual fixes came up while rewording: - The table formats section said Comet has no Delta Lake integration. main now has an experimental, opt-in Delta Lake scan (apache#5365), so say that and link the Delta Lake guide. - Link the compatibility guide by its path in the docs rather than by its published URL, and point the two Gluten limitations links, which return 404, at the page in Gluten's v1.7.0 sources. Part of apache#4419. Generated-by: OpenCode * docs: finish the apache#4419 terminology in Understanding Comet Plans apache#4650 rewrote this page. Fix the uses of "native" that remain: say "Rust" where the text means the Rust implementation or the Rust plan, "in the Comet pipeline" where it means Comet as a whole, and "Spark's built-in shuffle" instead of "Spark's native shuffle". Config keys, API names, and quoted explain output are unchanged. Also say that consecutive Rust-implemented operators are usually, not always, combined into one block, and add CometTakeOrderedAndProject to the exceptions: it runs its own final Rust plan, after a single-partition shuffle when its input has several partitions. Two small fixes on the same page: the explain section documents seven configs, not five, so add the missing spark.comet.explain.planOnly.enabled and spark.comet.explain.rules rows to its table; and CometCollectLimit is not purely JVM-side, since it runs Rust limit plans unless Spark collects it at the plan root, so say so and adjust the JVM-side section's intro. prettier realigns the Scans table because two of its rows changed. Part of apache#4419. Generated-by: OpenCode * docs: replace "DataFusion native engine" on the docs landing page Use the apache#4419 terminology on docs/source/index.md, which the README rewrite in apache#4428 did not cover. The page now says what Comet does with DataFusion rather than calling it a native engine, and the opening line carries the README's Arrow framing, as apache#4419 asks. Say that Comet hands DataFusion the operators it runs in Rust, not all supported operators, since some run on the JVM. Also drop "powerful" from the edited paragraph, and link the benchmarking guide by its relative path rather than its published URL. Part of apache#4419. Generated-by: OpenCode * docs: qualify direct read, explain.rules, and shuffle child rules - plugin_overview.md: a shuffle input is a ShuffleScanExec only when it is serialized as a ShuffleScan leaf, which needs direct read, an AQE query stage, and supported output types. Without AQE, the shuffle is read through ScanExec. - understanding-comet-plans.md: spark.comet.explain.rules logs the plan before and after CometScanRule, CometExecRule and EliminateRedundantTransitions, not stage reverts. Point to spark.comet.explain.fallback.log.enabled for those. - jvm_shuffle.md: scope the columnar-to-row diagram to children that produce Arrow batches. A CometSparkToColumnarExec child returns its input rows directly. - native_shuffle.md: with the Celeborn shuffle manager, native shuffle needs a CometNativeExec child, not any CometPlan. Part of apache#4419. Generated-by: OpenCode * docs: match the shuffle read condition, drop native and seamless from the home page The Shuffle Reads paragraph in plugin_overview.md now gives the same ShuffleScan condition as the paragraph above it and jvm_shuffle.md. The overview figure and the index.md compatibility blurb write Rust instead of native (N3) and drop seamlessly (P10). Generated-by: OpenCode * docs: say what decides the CometCollectLimit path, fix two overview figure labels CometCollectLimit takes rows on the JVM for collect, show and take, through executeCollect, and runs Rust limit plans otherwise; where it sits in the plan does not decide this. The overview figure no longer says every supported operator runs in Rust (CometUnion does not), and drops vectorized as a synonym for columnar (T10). Generated-by: OpenCode * docs: give the full Celeborn shuffle condition, address the reader on the Gluten page With the Celeborn shuffle manager, native shuffle also needs spark.comet.shuffle.mode=native and the manager's own support checks (cometCelebornShuffleFallbackReason), not only a CometNativeExec child. Two changed lines on the Gluten page now address the reader (P9). Generated-by: OpenCode * docs: make the native rules testable N1 now covers every form of the banned phrases, such as ran natively and Comet-native. N2 allows the native shuffle path without pairing it with JVM shuffle. N3 replaces and similar with a test, names user-guide/, and says pages outside the contributor guide allow only the N2 compounds. Agents running the guide could not apply the old wording consistently. Generated-by: OpenCode * docs: complete the CometRule and Celeborn conditions, say Rust plan for a fused block Under Adaptive Query Execution, CometRule also runs as the columnar rule on each new query stage, not only as a stage preparation rule. The Celeborn condition names nativeShuffleFallbackReason, which holds the manager's checks, instead of a user-guide section that only covers released clients. The pages already say Rust plan for a group of operators that run together in Rust, so the three Comet block mentions use it too. Generated-by: OpenCode * docs: allow Arrow-native in N2 and close the native rules T7 lists Arrow-native, which N3's limit to N2 compounds outside the contributor guide would otherwise forbid. N3 also says any other use of native is a finding, so a phrase outside the lists has an outcome. Generated-by: OpenCode * docs: limit the N2 scan compounds to Rust-implemented scans A dry run read native cache scan as allowed by N2, but the in-memory cache scan runs on the JVM. N2 now says the scan and writer compounds apply only to Rust-implemented ones. Generated-by: OpenCode * docs: point N1 at T1 for not-a-fallback and at N5 for Spark A dry run rewrote "falls back to its native execution" and "executes natively (not falling back to Spark)" wrongly, because N1 offered only T1 or T3. N1 now says not falling back is T1, since codegen dispatch is not a fallback, and that Spark's own behavior is N5. Generated-by: OpenCode * docs: apply the style guide to new and changed text Existing pages move over as they are edited, so a review does not ask a change to restyle text it leaves alone, unless the change makes that text wrong (P16). Requested in review. Generated-by: OpenCode * docs: call Velox and DataFusion vectorized query engines again The sentence is about batch-at-a-time execution, which T10 allows vectorized for, and both projects describe themselves that way. Columnar is a claim about data layout. Requested in review. Generated-by: OpenCode

andygrove marked this pull request as draft May 25, 2026 17:29

andygrove force-pushed the docs-nomenclature-4419 branch from f84408c to 9fbe7ce Compare May 28, 2026 14:10

andygrove force-pushed the docs-nomenclature-4419 branch from 9fbe7ce to 91420db Compare May 28, 2026 14:12

andygrove changed the title ~~docs: adopt Arrow-native nomenclature across user and contributor guides~~ docs: lead README with the Arrow-native framing May 28, 2026

andygrove marked this pull request as ready for review May 28, 2026 14:13

mbutrovich self-requested a review May 29, 2026 19:43

mbutrovich reviewed May 29, 2026

View reviewed changes

Comment thread README.md Resolved

mbutrovich reviewed May 29, 2026

View reviewed changes

andygrove and others added 2 commits June 1, 2026 08:05

Update README.md

235f4a5

Co-authored-by: Matt Butrovich <[email protected]>

docs: drop 'experimental' from Scala/Java UDF bullet

7f03711

apache#4514 (JVM Scala UDF codegen dispatch enabled by default) has merged, so the support is no longer experimental.

andygrove added this to the 0.17.0 milestone Jun 4, 2026

comphead reviewed Jun 10, 2026

View reviewed changes

Comment thread README.md Resolved

comphead reviewed Jun 10, 2026

View reviewed changes

Update README.md

a730b29

Co-authored-by: Oleks V <[email protected]>

comphead approved these changes Jun 10, 2026

View reviewed changes

andygrove merged commit 00897cc into apache:main Jun 10, 2026
15 checks passed

andygrove deleted the docs-nomenclature-4419 branch June 10, 2026 23:05

comphead pushed a commit to comphead/arrow-datafusion-comet that referenced this pull request Aug 26, 2026

docs: lead README with the Arrow-native framing (apache#4428)

2590462

This was referenced Oct 22, 2026

docs: add a documentation style guide and apply it to the remaining pages #7151

Merged

docs: update the plugin overview for current scans and #4419 terminology #7152

Closed

docs: replace "DataFusion native engine" on the docs landing page #7160

Closed

Navigation Menu

Repository navigation

docs: lead README with the Arrow-native framing - #4428

docs: lead README with the Arrow-native framing#4428
andygrove merged 4 commits into
apache:mainfrom
andygrove:docs-nomenclature-4419

andygrove commented May 25, 2026 •

edited

Loading

Uh oh!

Uh oh!

mbutrovich May 29, 2026

Uh oh!

andygrove Jun 1, 2026

Uh oh!

mbutrovich May 29, 2026

Uh oh!

Uh oh!

comphead Jun 10, 2026

Uh oh!

andygrove Jun 10, 2026

Uh oh!

comphead Jun 10, 2026

Uh oh!

comphead left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

andygrove commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

Uh oh!

mbutrovich May 29, 2026

Choose a reason for hiding this comment

Uh oh!

andygrove Jun 1, 2026

Choose a reason for hiding this comment

Uh oh!

mbutrovich May 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

comphead Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

andygrove Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

comphead Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andygrove commented May 25, 2026 •

edited

Loading