Skip to content

perf: benchmark task startup overlap - #873

Merged
gabotechs merged 2 commits into
datafusion-contrib:mainfrom
joybestourous:joy/codex/startup-benchmark
Nov 10, 2026
Merged

gabotechs merged 2 commits into
datafusion-contrib:mainfrom
joybestourous:joy/codex/startup-benchmark

Conversation

@joybestourous

@joybestourous joybestourous commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Task setup skew can postpone useful work on a ready worker. Add a Criterion benchmark for that scheduling case before the execution changes in #852. This PR changes no library behavior and adds no dependencies.

The benchmark runs one CPU-heavy task and one light task over localhost gRPC, in both static and AQE modes. Each mode has three cases: no routing delay, a 250 ms routing delay on the light task, and the same delay on the heavy task. It measures execution through complete output collection, excluding worker startup and planning. Every iteration checks output [0, 1] and exactly one heavy remote execution. An untimed preflight reports whether CPU work starts and finishes before delayed sibling routing returns.

Applied unchanged to #852, the benchmark demonstrates the conditional benefit:

Mode Case Baseline With #852 Paired change
Static No delay 228.15 ms 227.84 ms -0.14%
Static Delayed light task 479.29 ms 294.10 ms -38.64%
Static Delayed heavy task 479.09 ms 478.60 ms -0.10%
AQE No delay 228.23 ms 227.86 ms -0.16%
AQE Delayed light task 479.37 ms 294.36 ms -38.60%
AQE Delayed heavy task 479.52 ms 478.79 ms -0.15%

In both delayed-light preflights, heavy work started and finished before sibling routing returned with #852; neither witness occurred on the baseline. The controls stayed essentially flat. This demonstrates value when ready useful work can overlap a sibling's setup delay; it does not predict representative S3 or multi-host speedups.

Validation: optimized builds and all six cases passed on baseline and candidate. Formatting and Criterion-target Clippy with warnings denied passed.

Measurement scope and reproduction

Measurements used baseline 0291e08 (main e5d6ea3 plus this benchmark) and candidate source matching #852 at 354d386. The benchmark and lockfile were identical. The candidate binary was built before the final commit; source hashes were verified against that commit.

This was a controlled synthetic experiment on a shared host, with the same four-CPU affinity for both binaries, four Tokio threads, ten Criterion samples per case, one-second warmup and a five-second target measurement window. Three outer pairs were attempted in baseline/candidate, candidate/baseline, baseline/candidate order. Two opposite-order pairs passed the predefined resource gate. The entire third pair was excluded because baseline selected-CPU average utilization p95 was 85.31%, exceeding the 85% threshold; the maximum-utilization threshold was 98%. Raw results for both sides of the excluded pair were retained. Results were not filtered by measured latency.

Times above are medians of the accepted per-run Criterion medians; changes are medians of the paired percentage changes. Two outer pairs illustrate this scheduling case, rather than establishing production performance or precise population estimates.

cargo bench -p datafusion-distributed-benchmarks --bench startup_overlap -- --save-baseline main
# Run the same benchmark on the candidate revision:
cargo bench -p datafusion-distributed-benchmarks --bench startup_overlap -- --baseline main

Use separate build directories or rebuild both local packages when switching revisions. Confirm the candidate's delayed-light preflight shows early work and the baseline's does not before interpreting timings.

Measure completed queries with a delayed light task, a delayed heavy
task, and no setup delay in static and adaptive execution.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-6 <noreply@localhost>
@joybestourous
joybestourous marked this pull request as ready for review November 10, 2026 16:14

@gabotechs gabotechs left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! just a small comment about doc placement, but otherwise looks good. Thanks @joybestourous!

Move the microbenchmark instructions from the README into the benchmark
module documentation.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-6 <noreply@localhost>
@gabotechs
gabotechs merged commit cb6c9c4 into datafusion-contrib:main Nov 10, 2026
joybestourous added a commit to joybestourous/datafusion-distributed that referenced this pull request Nov 10, 2026
Keep the benchmark documentation beside its source, as merged in datafusion-contrib#873,
and remove the duplicate README section from this branch.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-6 <noreply@localhost>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants