Repository navigation
docs: present Ballista as one engine for ETL and interactive queries - #2834
Conversation
With pipelined execution merged, one cluster can run long ETL jobs staged and low-latency analytical queries pipelined, chosen per session. Say so in the README: the Features bullet notes that the two modes share a cluster, and a new section explains the lakehouse use case and the capacity a shared pool can save. The section also records what is missing: pipelined execution is experimental, and the scheduler has no job priority, so an interactive query can wait for vcores held by batch jobs.
github-actions
Bot
added
the
documentation
|
@villebro @milenkovicm I created this PR based on feedback that you both gave me in the past 24 hours |
milenkovicm
left a comment
milenkovicm
left a comment
There was a problem hiding this comment.
one engine to rule them all 😄 thanks @andygrove
villebro
left a comment
villebro
left a comment
There was a problem hiding this comment.
Being able to share infra across batch and interactive workloads with full control of workload priority would indeed be a game changer 🚀 Looks generally good, with a comment about calling out a single engine for interactive workloads.
| A lakehouse aims to merge the data warehouse and the ETL platform, but it is often still served by two engines: Spark | ||
| for batch ETL and Trino for interactive analytics. Ballista's two execution modes let one engine serve both, from the |
There was a problem hiding this comment.
While Spark can be considered the de facto leader in batch ETL, and Trino is certainly well established in enterprise settings, the interactive analytics engine landscape is arguably more fragmented (ClickHouse, StarRocks, Apache Doris/Pinot/Druid etc. come to mind).
Should we consider making this section more agnostic to specific reference engines, and instead describe the workload categories themselves? This would also future-proof the documentation as the landscape evolves and new contenders emerge in either space.
There was a problem hiding this comment.
Good point, thanks. I reworded the section to describe the two workload categories (long ETL jobs that need to survive task failures, and short analytical queries that need low latency) instead of naming Spark and Trino.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Which issue does this PR close?
N/A
Rationale for this change
With pipelined execution (#2657) merged, one Ballista cluster can run long ETL jobs and low-latency analytical queries side by side, but the README describes the two modes without saying what they add up to. A lakehouse aims to merge the data warehouse and the ETL platform, yet it is usually served by two engines: Spark for batch ETL and Trino for interactive analytics. Ballista can cover both, and a shared pool of executors can take less capacity than two clusters, because the two loads are often uncorrelated.
What changes are included in this PR?
README
The section also says what is missing, rather than promising it. Pipelined execution is experimental, and the scheduler has no job priority, so on a busy cluster an interactive query can wait for vcores held by batch jobs. The pipelined execution guide already lists this under Limitations, and #2652 listed fair admission between pipelined and staged jobs as a follow-up outside the first PR. No open issue tracks it.
What the claims rest on:
one_session_switches_between_staged_and_pipelined(ballista/client/tests/pipelined.rs) runs both modes on one cluster with identical results.bind_tasks_with), and a pipelined job reserves its vcores all at once, so nothing ranks one job above another.#2814 (draft) also edits the README, in the introduction and a new Performance section. A dry-run merge of the two is clean.
Are there any user-facing changes?
Documentation only. No code, API or wire-format changes.