Repository navigation
feat: upgrade to DataFusion 56 - #851
Rich-T-kid wants to merge 7 commits into
Conversation
Rich-T-kid
changed the title
Rich-T-kid
changed the title
|
benchmarks run tpch/sf1 |
|
benchmarks run tpch/sf10 |
|
benchmarks run tpch/sf100 |
|
Requested by this comment. Benchmark job 234 failed for How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
|
Requested by this comment. Benchmark job 235 failed for How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
|
Requested by this comment. Benchmark job 236 failed for How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
43bbdfb to
56e4baf
Compare
|
benchmarks run tpch/sf1 |
|
benchmarks run tpch/sf10 |
|
benchmarks run tpch/sf100 |
|
Requested by this comment. Benchmark job 237 failed for How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
|
Requested by this comment. Benchmark job 238 failed for How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
|
Requested by this comment. Benchmark job 239 failed for How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
56e4baf to
6486c6b
Compare
|
benchmarks run tpch/sf1 |
|
benchmarks run tpch/sf10 |
|
benchmarks run tpch/sf100 |
|
Requested by this comment. Benchmark job 240 failed for How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
|
Requested by this comment. Benchmark job 241 failed for How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
|
Requested by this comment. Benchmark job 242 failed for How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
Pin the unreleased DataFusion 56 branch, move the workspace to Arrow and Parquet 60, and update the benchmark dependency set. Use temporary Arrow 60-compatible Iceberg fork revisions until the corresponding upstream changes land.
6486c6b to
229c9c6
Compare
|
Requested by this comment. Benchmark job 243 failed for How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
Rich-T-kid
changed the title
|
Heads up on the benchmark failures: they're coming from the remote worker not compiling with this PR.
Bumping that line to |
229c9c6 to
ddc99f4
Compare
|
never mind on the iceberg failure, I think I initially had the dependency wrong. CI looks good |
ddc99f4 to
e5631a1
Compare
|
|
||
| ## 7. Upgrade DataFusion and Arrow dependencies |
There was a problem hiding this comment.
We should release 5.0.0 before the DF upgrade, so this might go away into a new 6.0.0.md
There was a problem hiding this comment.
ah okay, when is the intended release date for v5.0.0?
There was a problem hiding this comment.
next week, still dont know when exactly
DataFusion 56 uses object_store 0.14, so enabling the aws feature on object_store 0.13 does not affect DataFusion's re-export. Align the benchmark worker dependency so its AWS client compiles against the same package version.
Update TableProvider projections to borrowed slices, propagate fallible benchmark batches, and populate the new physical binary expression field. Disambiguate DFD byte metrics from the new DataFusion MetricBuilder method so the existing custom metric representation is preserved.
Record the DataFusion, Arrow, and Parquet compatibility requirements and the TableProvider projection signature change for downstream users.
Refresh Iceberg and distributed integration expectations for revised filters, limits, ordering metadata, and Parquet output size.
Update the planner, TPC-H, sorted TPC-H, and TPC-DS snapshots for DataFusion 56 optimizer and display changes.
e5631a1 to
23bfd3e
Compare
|
benchmarks run tpch/sf1 tpch/sf10 tpch/sf100 |
|
Requested by this comment. Benchmark resultsCompared: PR base === Comparing tpch_parquet/scale_factor=1/max_file_size=134217728/dictionary=pages/sorted=false results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] === TASKS: prev=139.0, new=132.0, diff=7.0 fewer (5.0%) (sum of per-query averages) TOTAL: prev=10270 ms, new=10310 ms, diff=1.00 slower ✖ Show full query output q1: prev= 434 ms, new= 435 ms, diff=1.00 slower ✖, tasks: prev=4.0, new=4.0, diff=no change
q2: prev= 278 ms, new= 287 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
q3: prev= 444 ms, new= 536 ms, diff=1.21 slower ✖, tasks: prev=5.0, new=5.0, diff=no change
q4: prev= 273 ms, new= 282 ms, diff=1.03 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
q5: prev= 546 ms, new= 483 ms, diff=1.13 faster ✔, tasks: prev=10.0, new=8.0, diff=2.0 fewer (20.0%)
q6: prev= 322 ms, new= 235 ms, diff=1.37 faster ✔, tasks: prev=2.0, new=2.0, diff=no change
q7: prev= 521 ms, new= 611 ms, diff=1.17 slower ✖, tasks: prev=13.0, new=11.0, diff=2.0 fewer (15.4%)
q8: prev= 651 ms, new= 690 ms, diff=1.06 slower ✖, tasks: prev=15.0, new=13.0, diff=2.0 fewer (13.3%)
q9: prev= 788 ms, new= 885 ms, diff=1.12 slower ✖, tasks: prev=13.0, new=13.0, diff=no change
q10: prev= 776 ms, new= 736 ms, diff=1.05 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
q11: prev= 234 ms, new= 247 ms, diff=1.06 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
q12: prev= 342 ms, new= 330 ms, diff=1.04 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
q13: prev= 707 ms, new= 775 ms, diff=1.10 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
q14: prev= 273 ms, new= 373 ms, diff=1.37 slower ✖, tasks: prev=5.0, new=4.0, diff=1.0 fewer (20.0%)
q15: prev= 470 ms, new= 430 ms, diff=1.09 faster ✔, tasks: prev=10.0, new=10.0, diff=no change
q16: prev= 189 ms, new= 150 ms, diff=1.26 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q17: prev= 612 ms, new= 490 ms, diff=1.25 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
q18: prev= 689 ms, new= 742 ms, diff=1.08 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
q19: prev= 295 ms, new= 297 ms, diff=1.01 slower ✖, tasks: prev=3.0, new=3.0, diff=no change
q20: prev= 529 ms, new= 521 ms, diff=1.02 faster ✔, tasks: prev=8.0, new=10.0, diff=2.0 more (25.0%)
q21: prev= 755 ms, new= 631 ms, diff=1.20 faster ✔, tasks: prev=17.0, new=15.0, diff=2.0 fewer (11.8%)
q22: prev= 142 ms, new= 144 ms, diff=1.01 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
=== Comparing tpch_parquet/scale_factor=10/max_file_size=134217728/dictionary=pages/sorted=false results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] === TASKS: prev=915.0, new=818.0, diff=97.0 fewer (10.6%) (sum of per-query averages) TOTAL: prev=20040 ms, new=19112 ms, diff=1.05 faster ✔ Show full query output q1: prev= 493 ms, new= 461 ms, diff=1.07 faster ✔, tasks: prev=24.0, new=24.0, diff=no change
q2: prev= 923 ms, new= 875 ms, diff=1.05 faster ✔, tasks: prev=37.0, new=32.0, diff=5.0 fewer (13.5%)
q3: prev= 907 ms, new=1108 ms, diff=1.22 slower ✖, tasks: prev=35.0, new=30.0, diff=5.0 fewer (14.3%)
q4: prev= 510 ms, new= 552 ms, diff=1.08 slower ✖, tasks: prev=41.0, new=41.0, diff=no change
q5: prev= 878 ms, new=1139 ms, diff=1.30 slower ✖, tasks: prev=62.0, new=50.0, diff=12.0 fewer (19.4%)
q6: prev= 360 ms, new= 419 ms, diff=1.16 slower ✖, tasks: prev=12.0, new=12.0, diff=no change
q7: prev= 853 ms, new= 966 ms, diff=1.13 slower ✖, tasks: prev=69.0, new=57.0, diff=12.0 fewer (17.4%)
q8: prev=1217 ms, new=1079 ms, diff=1.13 faster ✔, tasks: prev=83.0, new=59.0, diff=24.0 fewer (28.9%)
q9: prev=1634 ms, new=1158 ms, diff=1.41 faster ✔, tasks: prev=85.0, new=61.0, diff=24.0 fewer (28.2%)
q10: prev=2348 ms, new=2458 ms, diff=1.05 slower ✖, tasks: prev=48.0, new=48.0, diff=no change
q11: prev= 623 ms, new= 689 ms, diff=1.11 slower ✖, tasks: prev=30.0, new=20.0, diff=10.0 fewer (33.3%)
q12: prev= 529 ms, new= 531 ms, diff=1.00 slower ✖, tasks: prev=41.0, new=41.0, diff=no change
q13: prev= 822 ms, new= 855 ms, diff=1.04 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
q14: prev= 814 ms, new= 494 ms, diff=1.65 faster ✅, tasks: prev=25.0, new=25.0, diff=no change
q15: prev= 719 ms, new= 731 ms, diff=1.02 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
q16: prev= 513 ms, new= 456 ms, diff=1.13 faster ✔, tasks: prev=22.0, new=22.0, diff=no change
q17: prev= 881 ms, new= 758 ms, diff=1.16 faster ✔, tasks: prev=37.0, new=37.0, diff=no change
q18: prev=1554 ms, new=1311 ms, diff=1.19 faster ✔, tasks: prev=47.0, new=47.0, diff=no change
q19: prev= 535 ms, new= 467 ms, diff=1.15 faster ✔, tasks: prev=25.0, new=25.0, diff=no change
q20: prev=1273 ms, new=1082 ms, diff=1.18 faster ✔, tasks: prev=48.0, new=43.0, diff=5.0 fewer (10.4%)
q21: prev=1188 ms, new=1034 ms, diff=1.15 faster ✔, tasks: prev=67.0, new=67.0, diff=no change
q22: prev= 466 ms, new= 489 ms, diff=1.05 slower ✖, tasks: prev=11.0, new=11.0, diff=no change
=== Comparing tpch_parquet/scale_factor=100/max_file_size=134217728/dictionary=pages/sorted=false results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] === TASKS: prev=1320.0, new=1320.0, diff=no change (sum of per-query averages) TOTAL: prev=59678 ms, new=65658 ms, diff=1.10 slower ❌ Show full query output q1: prev=1744 ms, new=1873 ms, diff=1.07 slower ✖, tasks: prev=24.0, new=24.0, diff=no change
q2: prev=3048 ms, new=3392 ms, diff=1.11 slower ✖, tasks: prev=96.0, new=96.0, diff=no change
q3: prev=2671 ms, new=2520 ms, diff=1.06 faster ✔, tasks: prev=58.0, new=58.0, diff=no change
q4: prev=1113 ms, new=1492 ms, diff=1.34 slower ✖, tasks: prev=48.0, new=48.0, diff=no change
q5: prev=2619 ms, new=3056 ms, diff=1.17 slower ✖, tasks: prev=85.0, new=85.0, diff=no change
q6: prev=1041 ms, new= 930 ms, diff=1.12 faster ✔, tasks: prev=12.0, new=12.0, diff=no change
q7: prev=3002 ms, new=2990 ms, diff=1.00 faster ✔, tasks: prev=85.0, new=85.0, diff=no change
q8: prev=3741 ms, new=3788 ms, diff=1.01 slower ✖, tasks: prev=104.0, new=104.0, diff=no change
q9: prev=4998 ms, new=4947 ms, diff=1.01 faster ✔, tasks: prev=104.0, new=104.0, diff=no change
q10: prev=5136 ms, new=5634 ms, diff=1.10 slower ✖, tasks: prev=71.0, new=71.0, diff=no change
q11: prev=1034 ms, new=3515 ms, diff=3.40 slower ❌, tasks: prev=65.0, new=65.0, diff=no change
q12: prev=1446 ms, new=2172 ms, diff=1.50 slower ❌, tasks: prev=48.0, new=48.0, diff=no change
q13: prev=1680 ms, new=2187 ms, diff=1.30 slower ✖, tasks: prev=46.0, new=46.0, diff=no change
q14: prev=1337 ms, new=1332 ms, diff=1.00 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
q15: prev=3390 ms, new=5051 ms, diff=1.49 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
q16: prev=2439 ms, new=1606 ms, diff=1.52 faster ✅, tasks: prev=55.0, new=55.0, diff=no change
q17: prev=4314 ms, new=3399 ms, diff=1.27 faster ✔, tasks: prev=42.0, new=42.0, diff=no change
q18: prev=4314 ms, new=4451 ms, diff=1.03 slower ✖, tasks: prev=70.0, new=70.0, diff=no change
q19: prev=1418 ms, new=1549 ms, diff=1.09 slower ✖, tasks: prev=30.0, new=30.0, diff=no change
q20: prev=3141 ms, new=3746 ms, diff=1.19 slower ✖, tasks: prev=67.0, new=67.0, diff=no change
q21: prev=5226 ms, new=5119 ms, diff=1.02 faster ✔, tasks: prev=86.0, new=86.0, diff=no change
q22: prev= 826 ms, new= 909 ms, diff=1.10 slower ✖, tasks: prev=44.0, new=44.0, diff=no change
Verification and run detailsJob
Workload: Capacity: 12 Other timings: Queue 1s · Dataset validation 1s · Total 26m 44s How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
|
benchmarks run tpch/sf1 tpch/sf10 tpch/sf100 |
2 similar comments
|
benchmarks run tpch/sf1 tpch/sf10 tpch/sf100 |
|
benchmarks run tpch/sf1 tpch/sf10 tpch/sf100 |
|
Requested by this comment. Benchmark resultsCompared: PR base === Comparing tpch_parquet/scale_factor=1/max_file_size=134217728/dictionary=pages/sorted=false results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] === TASKS: prev=139.0, new=132.0, diff=7.0 fewer (5.0%) (sum of per-query averages) TOTAL: prev=10788 ms, new=9521 ms, diff=1.13 faster ✅ Show full query output q1: prev= 470 ms, new= 415 ms, diff=1.13 faster ✔, tasks: prev=4.0, new=4.0, diff=no change
q2: prev= 259 ms, new= 266 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
q3: prev= 572 ms, new= 451 ms, diff=1.27 faster ✔, tasks: prev=5.0, new=5.0, diff=no change
q4: prev= 278 ms, new= 271 ms, diff=1.03 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
q5: prev= 462 ms, new= 504 ms, diff=1.09 slower ✖, tasks: prev=10.0, new=8.0, diff=2.0 fewer (20.0%)
q6: prev= 224 ms, new= 208 ms, diff=1.08 faster ✔, tasks: prev=2.0, new=2.0, diff=no change
q7: prev= 574 ms, new= 495 ms, diff=1.16 faster ✔, tasks: prev=13.0, new=11.0, diff=2.0 fewer (15.4%)
q8: prev= 680 ms, new= 540 ms, diff=1.26 faster ✔, tasks: prev=15.0, new=13.0, diff=2.0 fewer (13.3%)
q9: prev= 790 ms, new= 669 ms, diff=1.18 faster ✔, tasks: prev=13.0, new=13.0, diff=no change
q10: prev= 742 ms, new= 705 ms, diff=1.05 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
q11: prev= 240 ms, new= 271 ms, diff=1.13 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
q12: prev= 420 ms, new= 387 ms, diff=1.09 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
q13: prev= 874 ms, new= 721 ms, diff=1.21 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q14: prev= 421 ms, new= 287 ms, diff=1.47 faster ✔, tasks: prev=5.0, new=4.0, diff=1.0 fewer (20.0%)
q15: prev= 536 ms, new= 415 ms, diff=1.29 faster ✔, tasks: prev=10.0, new=10.0, diff=no change
q16: prev= 210 ms, new= 140 ms, diff=1.50 faster ✅, tasks: prev=0.0, new=0.0, diff=no change
q17: prev= 515 ms, new= 463 ms, diff=1.11 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
q18: prev= 747 ms, new= 745 ms, diff=1.00 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
q19: prev= 347 ms, new= 309 ms, diff=1.12 faster ✔, tasks: prev=3.0, new=3.0, diff=no change
q20: prev= 506 ms, new= 503 ms, diff=1.01 faster ✔, tasks: prev=8.0, new=10.0, diff=2.0 more (25.0%)
q21: prev= 757 ms, new= 630 ms, diff=1.20 faster ✔, tasks: prev=17.0, new=15.0, diff=2.0 fewer (11.8%)
q22: prev= 164 ms, new= 126 ms, diff=1.30 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
=== Comparing tpch_parquet/scale_factor=10/max_file_size=134217728/dictionary=pages/sorted=false results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] === TASKS: prev=915.0, new=818.0, diff=97.0 fewer (10.6%) (sum of per-query averages) TOTAL: prev=19758 ms, new=17490 ms, diff=1.13 faster ✅ Show full query output q1: prev= 465 ms, new= 406 ms, diff=1.15 faster ✔, tasks: prev=24.0, new=24.0, diff=no change
q2: prev= 810 ms, new= 877 ms, diff=1.08 slower ✖, tasks: prev=37.0, new=32.0, diff=5.0 fewer (13.5%)
q3: prev= 839 ms, new= 798 ms, diff=1.05 faster ✔, tasks: prev=35.0, new=30.0, diff=5.0 fewer (14.3%)
q4: prev= 430 ms, new= 398 ms, diff=1.08 faster ✔, tasks: prev=41.0, new=41.0, diff=no change
q5: prev= 924 ms, new= 903 ms, diff=1.02 faster ✔, tasks: prev=62.0, new=50.0, diff=12.0 fewer (19.4%)
q6: prev= 508 ms, new= 442 ms, diff=1.15 faster ✔, tasks: prev=12.0, new=12.0, diff=no change
q7: prev= 870 ms, new= 815 ms, diff=1.07 faster ✔, tasks: prev=69.0, new=57.0, diff=12.0 fewer (17.4%)
q8: prev=1128 ms, new= 874 ms, diff=1.29 faster ✔, tasks: prev=83.0, new=59.0, diff=24.0 fewer (28.9%)
q9: prev=1873 ms, new=1129 ms, diff=1.66 faster ✅, tasks: prev=85.0, new=61.0, diff=24.0 fewer (28.2%)
q10: prev=2455 ms, new=2405 ms, diff=1.02 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
q11: prev= 615 ms, new= 498 ms, diff=1.23 faster ✔, tasks: prev=30.0, new=20.0, diff=10.0 fewer (33.3%)
q12: prev= 476 ms, new= 520 ms, diff=1.09 slower ✖, tasks: prev=41.0, new=41.0, diff=no change
q13: prev= 885 ms, new= 846 ms, diff=1.05 faster ✔, tasks: prev=16.0, new=16.0, diff=no change
q14: prev= 608 ms, new= 433 ms, diff=1.40 faster ✔, tasks: prev=25.0, new=25.0, diff=no change
q15: prev= 670 ms, new= 645 ms, diff=1.04 faster ✔, tasks: prev=50.0, new=50.0, diff=no change
q16: prev= 458 ms, new= 433 ms, diff=1.06 faster ✔, tasks: prev=22.0, new=22.0, diff=no change
q17: prev= 835 ms, new= 725 ms, diff=1.15 faster ✔, tasks: prev=37.0, new=37.0, diff=no change
q18: prev=1305 ms, new=1301 ms, diff=1.00 faster ✔, tasks: prev=47.0, new=47.0, diff=no change
q19: prev= 509 ms, new= 457 ms, diff=1.11 faster ✔, tasks: prev=25.0, new=25.0, diff=no change
q20: prev=1233 ms, new=1078 ms, diff=1.14 faster ✔, tasks: prev=48.0, new=43.0, diff=5.0 fewer (10.4%)
q21: prev=1300 ms, new= 987 ms, diff=1.32 faster ✔, tasks: prev=67.0, new=67.0, diff=no change
q22: prev= 562 ms, new= 520 ms, diff=1.08 faster ✔, tasks: prev=11.0, new=11.0, diff=no change
=== Comparing tpch_parquet/scale_factor=100/max_file_size=134217728/dictionary=pages/sorted=false results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] === TASKS: prev=1320.0, new=1320.0, diff=no change (sum of per-query averages) TOTAL: prev=59635 ms, new=54119 ms, diff=1.10 faster ✅ Show full query output q1: prev=1762 ms, new=1697 ms, diff=1.04 faster ✔, tasks: prev=24.0, new=24.0, diff=no change
q2: prev=3701 ms, new=2890 ms, diff=1.28 faster ✔, tasks: prev=96.0, new=96.0, diff=no change
q3: prev=3565 ms, new=2266 ms, diff=1.57 faster ✅, tasks: prev=58.0, new=58.0, diff=no change
q4: prev=1594 ms, new=1041 ms, diff=1.53 faster ✅, tasks: prev=48.0, new=48.0, diff=no change
q5: prev=2940 ms, new=2666 ms, diff=1.10 faster ✔, tasks: prev=85.0, new=85.0, diff=no change
q6: prev= 983 ms, new= 887 ms, diff=1.11 faster ✔, tasks: prev=12.0, new=12.0, diff=no change
q7: prev=2918 ms, new=2883 ms, diff=1.01 faster ✔, tasks: prev=85.0, new=85.0, diff=no change
q8: prev=3748 ms, new=3529 ms, diff=1.06 faster ✔, tasks: prev=104.0, new=104.0, diff=no change
q9: prev=5167 ms, new=5039 ms, diff=1.03 faster ✔, tasks: prev=104.0, new=104.0, diff=no change
q10: prev=5890 ms, new=4912 ms, diff=1.20 faster ✔, tasks: prev=71.0, new=71.0, diff=no change
q11: prev=1262 ms, new= 950 ms, diff=1.33 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
q12: prev=1351 ms, new=1052 ms, diff=1.28 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
q13: prev=1206 ms, new=1199 ms, diff=1.01 faster ✔, tasks: prev=46.0, new=46.0, diff=no change
q14: prev=1278 ms, new=1181 ms, diff=1.08 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
q15: prev=3002 ms, new=3342 ms, diff=1.11 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
q16: prev=1277 ms, new=1357 ms, diff=1.06 slower ✖, tasks: prev=55.0, new=55.0, diff=no change
q17: prev=3840 ms, new=3287 ms, diff=1.17 faster ✔, tasks: prev=42.0, new=42.0, diff=no change
q18: prev=3736 ms, new=4171 ms, diff=1.12 slower ✖, tasks: prev=70.0, new=70.0, diff=no change
q19: prev=1571 ms, new=1284 ms, diff=1.22 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
q20: prev=3211 ms, new=2975 ms, diff=1.08 faster ✔, tasks: prev=67.0, new=67.0, diff=no change
q21: prev=4799 ms, new=4703 ms, diff=1.02 faster ✔, tasks: prev=86.0, new=86.0, diff=no change
q22: prev= 834 ms, new= 808 ms, diff=1.03 faster ✔, tasks: prev=44.0, new=44.0, diff=no change
Verification and run detailsJob
Workload: Capacity: 12 Other timings: Queue 5s · Dataset validation 1s · Total 22m 41s How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
|
Requested by this comment. Benchmark resultsCompared: PR base === Comparing tpch_parquet/scale_factor=1/max_file_size=134217728/dictionary=pages/sorted=false results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] === TASKS: prev=139.0, new=132.0, diff=7.0 fewer (5.0%) (sum of per-query averages) TOTAL: prev=9326 ms, new=9110 ms, diff=1.02 faster ✔ Show full query output q1: prev= 390 ms, new= 344 ms, diff=1.13 faster ✔, tasks: prev=4.0, new=4.0, diff=no change
q2: prev= 264 ms, new= 254 ms, diff=1.04 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q3: prev= 441 ms, new= 439 ms, diff=1.00 faster ✔, tasks: prev=5.0, new=5.0, diff=no change
q4: prev= 230 ms, new= 260 ms, diff=1.13 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
q5: prev= 432 ms, new= 390 ms, diff=1.11 faster ✔, tasks: prev=10.0, new=8.0, diff=2.0 fewer (20.0%)
q6: prev= 205 ms, new= 196 ms, diff=1.05 faster ✔, tasks: prev=2.0, new=2.0, diff=no change
q7: prev= 516 ms, new= 475 ms, diff=1.09 faster ✔, tasks: prev=13.0, new=11.0, diff=2.0 fewer (15.4%)
q8: prev= 597 ms, new= 503 ms, diff=1.19 faster ✔, tasks: prev=15.0, new=13.0, diff=2.0 fewer (13.3%)
q9: prev= 802 ms, new= 731 ms, diff=1.10 faster ✔, tasks: prev=13.0, new=13.0, diff=no change
q10: prev= 696 ms, new= 695 ms, diff=1.00 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
q11: prev= 251 ms, new= 229 ms, diff=1.10 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q12: prev= 343 ms, new= 305 ms, diff=1.12 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
q13: prev= 706 ms, new= 737 ms, diff=1.04 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
q14: prev= 277 ms, new= 331 ms, diff=1.19 slower ✖, tasks: prev=5.0, new=4.0, diff=1.0 fewer (20.0%)
q15: prev= 410 ms, new= 403 ms, diff=1.02 faster ✔, tasks: prev=10.0, new=10.0, diff=no change
q16: prev= 171 ms, new= 147 ms, diff=1.16 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q17: prev= 440 ms, new= 463 ms, diff=1.05 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
q18: prev= 619 ms, new= 698 ms, diff=1.13 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
q19: prev= 278 ms, new= 261 ms, diff=1.07 faster ✔, tasks: prev=3.0, new=3.0, diff=no change
q20: prev= 483 ms, new= 509 ms, diff=1.05 slower ✖, tasks: prev=8.0, new=10.0, diff=2.0 more (25.0%)
q21: prev= 644 ms, new= 612 ms, diff=1.05 faster ✔, tasks: prev=17.0, new=15.0, diff=2.0 fewer (11.8%)
q22: prev= 131 ms, new= 128 ms, diff=1.02 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
=== Comparing tpch_parquet/scale_factor=10/max_file_size=134217728/dictionary=pages/sorted=false results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] === TASKS: prev=915.0, new=818.0, diff=97.0 fewer (10.6%) (sum of per-query averages) TOTAL: prev=18172 ms, new=16896 ms, diff=1.08 faster ✔ Show full query output q1: prev= 391 ms, new= 506 ms, diff=1.29 slower ✖, tasks: prev=24.0, new=24.0, diff=no change
q2: prev= 916 ms, new= 768 ms, diff=1.19 faster ✔, tasks: prev=37.0, new=32.0, diff=5.0 fewer (13.5%)
q3: prev= 823 ms, new= 899 ms, diff=1.09 slower ✖, tasks: prev=35.0, new=30.0, diff=5.0 fewer (14.3%)
q4: prev= 567 ms, new= 524 ms, diff=1.08 faster ✔, tasks: prev=41.0, new=41.0, diff=no change
q5: prev= 770 ms, new= 806 ms, diff=1.05 slower ✖, tasks: prev=62.0, new=50.0, diff=12.0 fewer (19.4%)
q6: prev= 299 ms, new= 258 ms, diff=1.16 faster ✔, tasks: prev=12.0, new=12.0, diff=no change
q7: prev= 841 ms, new= 774 ms, diff=1.09 faster ✔, tasks: prev=69.0, new=57.0, diff=12.0 fewer (17.4%)
q8: prev=1109 ms, new= 919 ms, diff=1.21 faster ✔, tasks: prev=83.0, new=59.0, diff=24.0 fewer (28.9%)
q9: prev=1552 ms, new=1224 ms, diff=1.27 faster ✔, tasks: prev=85.0, new=61.0, diff=24.0 fewer (28.2%)
q10: prev=2576 ms, new=2325 ms, diff=1.11 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
q11: prev= 533 ms, new= 484 ms, diff=1.10 faster ✔, tasks: prev=30.0, new=20.0, diff=10.0 fewer (33.3%)
q12: prev= 422 ms, new= 389 ms, diff=1.08 faster ✔, tasks: prev=41.0, new=41.0, diff=no change
q13: prev= 822 ms, new= 831 ms, diff=1.01 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
q14: prev= 403 ms, new= 401 ms, diff=1.00 faster ✔, tasks: prev=25.0, new=25.0, diff=no change
q15: prev= 667 ms, new= 583 ms, diff=1.14 faster ✔, tasks: prev=50.0, new=50.0, diff=no change
q16: prev= 434 ms, new= 424 ms, diff=1.02 faster ✔, tasks: prev=22.0, new=22.0, diff=no change
q17: prev= 751 ms, new= 611 ms, diff=1.23 faster ✔, tasks: prev=37.0, new=37.0, diff=no change
q18: prev=1222 ms, new=1198 ms, diff=1.02 faster ✔, tasks: prev=47.0, new=47.0, diff=no change
q19: prev= 437 ms, new= 439 ms, diff=1.00 slower ✖, tasks: prev=25.0, new=25.0, diff=no change
q20: prev=1136 ms, new=1075 ms, diff=1.06 faster ✔, tasks: prev=48.0, new=43.0, diff=5.0 fewer (10.4%)
q21: prev=1029 ms, new= 982 ms, diff=1.05 faster ✔, tasks: prev=67.0, new=67.0, diff=no change
q22: prev= 472 ms, new= 476 ms, diff=1.01 slower ✖, tasks: prev=11.0, new=11.0, diff=no change
=== Comparing tpch_parquet/scale_factor=100/max_file_size=134217728/dictionary=pages/sorted=false results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] === TASKS: prev=1320.0, new=1320.0, diff=no change (sum of per-query averages) TOTAL: prev=57378 ms, new=57017 ms, diff=1.01 faster ✔ Show full query output q1: prev=1757 ms, new=1699 ms, diff=1.03 faster ✔, tasks: prev=24.0, new=24.0, diff=no change
q2: prev=3126 ms, new=2994 ms, diff=1.04 faster ✔, tasks: prev=96.0, new=96.0, diff=no change
q3: prev=2389 ms, new=2310 ms, diff=1.03 faster ✔, tasks: prev=58.0, new=58.0, diff=no change
q4: prev=1010 ms, new=1036 ms, diff=1.03 slower ✖, tasks: prev=48.0, new=48.0, diff=no change
q5: prev=2701 ms, new=2604 ms, diff=1.04 faster ✔, tasks: prev=85.0, new=85.0, diff=no change
q6: prev= 961 ms, new= 914 ms, diff=1.05 faster ✔, tasks: prev=12.0, new=12.0, diff=no change
q7: prev=2980 ms, new=4306 ms, diff=1.44 slower ✖, tasks: prev=85.0, new=85.0, diff=no change
q8: prev=3361 ms, new=4590 ms, diff=1.37 slower ✖, tasks: prev=104.0, new=104.0, diff=no change
q9: prev=4715 ms, new=4900 ms, diff=1.04 slower ✖, tasks: prev=104.0, new=104.0, diff=no change
q10: prev=5429 ms, new=4832 ms, diff=1.12 faster ✔, tasks: prev=71.0, new=71.0, diff=no change
q11: prev=1200 ms, new= 990 ms, diff=1.21 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
q12: prev=1324 ms, new=1101 ms, diff=1.20 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
q13: prev=1196 ms, new=1274 ms, diff=1.07 slower ✖, tasks: prev=46.0, new=46.0, diff=no change
q14: prev=1249 ms, new=1137 ms, diff=1.10 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
q15: prev=3433 ms, new=3215 ms, diff=1.07 faster ✔, tasks: prev=50.0, new=50.0, diff=no change
q16: prev=1775 ms, new=1682 ms, diff=1.06 faster ✔, tasks: prev=55.0, new=55.0, diff=no change
q17: prev=4478 ms, new=2876 ms, diff=1.56 faster ✅, tasks: prev=42.0, new=42.0, diff=no change
q18: prev=4128 ms, new=4133 ms, diff=1.00 slower ✖, tasks: prev=70.0, new=70.0, diff=no change
q19: prev=1559 ms, new=1300 ms, diff=1.20 faster ✔, tasks: prev=30.0, new=30.0, diff=no change
q20: prev=2906 ms, new=3124 ms, diff=1.08 slower ✖, tasks: prev=67.0, new=67.0, diff=no change
q21: prev=4901 ms, new=5167 ms, diff=1.05 slower ✖, tasks: prev=86.0, new=86.0, diff=no change
q22: prev= 800 ms, new= 833 ms, diff=1.04 slower ✖, tasks: prev=44.0, new=44.0, diff=no change
Verification and run detailsJob
Workload: Capacity: 12 Other timings: Queue 22m 44s · Dataset validation 1s · Total 21m 37s How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
|
Requested by this comment. Benchmark resultsCompared: PR base === Comparing tpch_parquet/scale_factor=1/max_file_size=134217728/dictionary=pages/sorted=false results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] === TASKS: prev=139.0, new=132.0, diff=7.0 fewer (5.0%) (sum of per-query averages) TOTAL: prev=9822 ms, new=9475 ms, diff=1.04 faster ✔ Show full query output q1: prev= 417 ms, new= 402 ms, diff=1.04 faster ✔, tasks: prev=4.0, new=4.0, diff=no change
q2: prev= 271 ms, new= 250 ms, diff=1.08 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q3: prev= 554 ms, new= 558 ms, diff=1.01 slower ✖, tasks: prev=5.0, new=5.0, diff=no change
q4: prev= 244 ms, new= 254 ms, diff=1.04 slower ✖, tasks: prev=7.0, new=7.0, diff=no change
q5: prev= 432 ms, new= 481 ms, diff=1.11 slower ✖, tasks: prev=10.0, new=8.0, diff=2.0 fewer (20.0%)
q6: prev= 206 ms, new= 233 ms, diff=1.13 slower ✖, tasks: prev=2.0, new=2.0, diff=no change
q7: prev= 535 ms, new= 494 ms, diff=1.08 faster ✔, tasks: prev=13.0, new=11.0, diff=2.0 fewer (15.4%)
q8: prev= 609 ms, new= 559 ms, diff=1.09 faster ✔, tasks: prev=15.0, new=13.0, diff=2.0 fewer (13.3%)
q9: prev= 756 ms, new= 747 ms, diff=1.01 faster ✔, tasks: prev=13.0, new=13.0, diff=no change
q10: prev= 742 ms, new= 677 ms, diff=1.10 faster ✔, tasks: prev=6.0, new=6.0, diff=no change
q11: prev= 237 ms, new= 237 ms, diff=1.00 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
q12: prev= 348 ms, new= 314 ms, diff=1.11 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
q13: prev= 697 ms, new= 717 ms, diff=1.03 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
q14: prev= 294 ms, new= 318 ms, diff=1.08 slower ✖, tasks: prev=5.0, new=4.0, diff=1.0 fewer (20.0%)
q15: prev= 476 ms, new= 416 ms, diff=1.14 faster ✔, tasks: prev=10.0, new=10.0, diff=no change
q16: prev= 166 ms, new= 149 ms, diff=1.11 faster ✔, tasks: prev=0.0, new=0.0, diff=no change
q17: prev= 495 ms, new= 455 ms, diff=1.09 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
q18: prev= 705 ms, new= 665 ms, diff=1.06 faster ✔, tasks: prev=7.0, new=7.0, diff=no change
q19: prev= 304 ms, new= 288 ms, diff=1.06 faster ✔, tasks: prev=3.0, new=3.0, diff=no change
q20: prev= 506 ms, new= 521 ms, diff=1.03 slower ✖, tasks: prev=8.0, new=10.0, diff=2.0 more (25.0%)
q21: prev= 699 ms, new= 609 ms, diff=1.15 faster ✔, tasks: prev=17.0, new=15.0, diff=2.0 fewer (11.8%)
q22: prev= 129 ms, new= 131 ms, diff=1.02 slower ✖, tasks: prev=0.0, new=0.0, diff=no change
=== Comparing tpch_parquet/scale_factor=10/max_file_size=134217728/dictionary=pages/sorted=false results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] === TASKS: prev=915.0, new=818.0, diff=97.0 fewer (10.6%) (sum of per-query averages) TOTAL: prev=18721 ms, new=20110 ms, diff=1.07 slower ✖ Show full query output q1: prev= 497 ms, new= 545 ms, diff=1.10 slower ✖, tasks: prev=24.0, new=24.0, diff=no change
q2: prev= 922 ms, new= 865 ms, diff=1.07 faster ✔, tasks: prev=37.0, new=32.0, diff=5.0 fewer (13.5%)
q3: prev= 994 ms, new= 891 ms, diff=1.12 faster ✔, tasks: prev=35.0, new=30.0, diff=5.0 fewer (14.3%)
q4: prev= 405 ms, new= 569 ms, diff=1.40 slower ✖, tasks: prev=41.0, new=41.0, diff=no change
q5: prev= 883 ms, new=1242 ms, diff=1.41 slower ✖, tasks: prev=62.0, new=50.0, diff=12.0 fewer (19.4%)
q6: prev= 325 ms, new= 754 ms, diff=2.32 slower ❌, tasks: prev=12.0, new=12.0, diff=no change
q7: prev= 889 ms, new= 986 ms, diff=1.11 slower ✖, tasks: prev=69.0, new=57.0, diff=12.0 fewer (17.4%)
q8: prev=1089 ms, new=1591 ms, diff=1.46 slower ✖, tasks: prev=83.0, new=59.0, diff=24.0 fewer (28.9%)
q9: prev=1582 ms, new=1388 ms, diff=1.14 faster ✔, tasks: prev=85.0, new=61.0, diff=24.0 fewer (28.2%)
q10: prev=2416 ms, new=2432 ms, diff=1.01 slower ✖, tasks: prev=48.0, new=48.0, diff=no change
q11: prev= 533 ms, new= 531 ms, diff=1.00 faster ✔, tasks: prev=30.0, new=20.0, diff=10.0 fewer (33.3%)
q12: prev= 441 ms, new= 443 ms, diff=1.00 slower ✖, tasks: prev=41.0, new=41.0, diff=no change
q13: prev= 822 ms, new= 850 ms, diff=1.03 slower ✖, tasks: prev=16.0, new=16.0, diff=no change
q14: prev= 500 ms, new= 474 ms, diff=1.05 faster ✔, tasks: prev=25.0, new=25.0, diff=no change
q15: prev= 568 ms, new= 694 ms, diff=1.22 slower ✖, tasks: prev=50.0, new=50.0, diff=no change
q16: prev= 414 ms, new= 550 ms, diff=1.33 slower ✖, tasks: prev=22.0, new=22.0, diff=no change
q17: prev= 711 ms, new= 618 ms, diff=1.15 faster ✔, tasks: prev=37.0, new=37.0, diff=no change
q18: prev=1195 ms, new=1239 ms, diff=1.04 slower ✖, tasks: prev=47.0, new=47.0, diff=no change
q19: prev= 536 ms, new= 478 ms, diff=1.12 faster ✔, tasks: prev=25.0, new=25.0, diff=no change
q20: prev=1281 ms, new=1100 ms, diff=1.16 faster ✔, tasks: prev=48.0, new=43.0, diff=5.0 fewer (10.4%)
q21: prev=1189 ms, new=1272 ms, diff=1.07 slower ✖, tasks: prev=67.0, new=67.0, diff=no change
q22: prev= 529 ms, new= 598 ms, diff=1.13 slower ✖, tasks: prev=11.0, new=11.0, diff=no change
=== Comparing tpch_parquet/scale_factor=100/max_file_size=134217728/dictionary=pages/sorted=false results 'datafusion-benchmark-base' [prev] with 'datafusion-benchmark-head' [new] === TASKS: prev=1320.0, new=1320.0, diff=no change (sum of per-query averages) TOTAL: prev=58756 ms, new=56777 ms, diff=1.03 faster ✔ Show full query output q1: prev=1763 ms, new=1800 ms, diff=1.02 slower ✖, tasks: prev=24.0, new=24.0, diff=no change
q2: prev=3152 ms, new=2885 ms, diff=1.09 faster ✔, tasks: prev=96.0, new=96.0, diff=no change
q3: prev=2545 ms, new=2410 ms, diff=1.06 faster ✔, tasks: prev=58.0, new=58.0, diff=no change
q4: prev=1231 ms, new=1125 ms, diff=1.09 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
q5: prev=3014 ms, new=2696 ms, diff=1.12 faster ✔, tasks: prev=85.0, new=85.0, diff=no change
q6: prev=1159 ms, new= 926 ms, diff=1.25 faster ✔, tasks: prev=12.0, new=12.0, diff=no change
q7: prev=3313 ms, new=2915 ms, diff=1.14 faster ✔, tasks: prev=85.0, new=85.0, diff=no change
q8: prev=3938 ms, new=3546 ms, diff=1.11 faster ✔, tasks: prev=104.0, new=104.0, diff=no change
q9: prev=5787 ms, new=4726 ms, diff=1.22 faster ✔, tasks: prev=104.0, new=104.0, diff=no change
q10: prev=5407 ms, new=5248 ms, diff=1.03 faster ✔, tasks: prev=71.0, new=71.0, diff=no change
q11: prev=1089 ms, new=1014 ms, diff=1.07 faster ✔, tasks: prev=65.0, new=65.0, diff=no change
q12: prev=1346 ms, new=1206 ms, diff=1.12 faster ✔, tasks: prev=48.0, new=48.0, diff=no change
q13: prev=1300 ms, new=1270 ms, diff=1.02 faster ✔, tasks: prev=46.0, new=46.0, diff=no change
q14: prev=1284 ms, new=1433 ms, diff=1.12 slower ✖, tasks: prev=30.0, new=30.0, diff=no change
q15: prev=3239 ms, new=3053 ms, diff=1.06 faster ✔, tasks: prev=50.0, new=50.0, diff=no change
q16: prev=1410 ms, new=1395 ms, diff=1.01 faster ✔, tasks: prev=55.0, new=55.0, diff=no change
q17: prev=3762 ms, new=3040 ms, diff=1.24 faster ✔, tasks: prev=42.0, new=42.0, diff=no change
q18: prev=3827 ms, new=4400 ms, diff=1.15 slower ✖, tasks: prev=70.0, new=70.0, diff=no change
q19: prev=1277 ms, new=1397 ms, diff=1.09 slower ✖, tasks: prev=30.0, new=30.0, diff=no change
q20: prev=2965 ms, new=3339 ms, diff=1.13 slower ✖, tasks: prev=67.0, new=67.0, diff=no change
q21: prev=5098 ms, new=5830 ms, diff=1.14 slower ✖, tasks: prev=86.0, new=86.0, diff=no change
q22: prev= 850 ms, new=1123 ms, diff=1.32 slower ✖, tasks: prev=44.0, new=44.0, diff=no change
Verification and run detailsJob
Workload: Capacity: 12 Other timings: Queue 1s · Dataset validation 1s · Total 22m 30s How to use the benchmark botPost a comment whose first non-empty line is:
For example:
Request one or more datasets, without duplicates. The bot checks each one against the dataset bucket before provisioning anything and runs every query in each dataset. Short names predating the parameter-encoded layout, such as
Supported instances and per-worker limits:
Each node runs one worker. The worker allocation reserves 1 vCPU and 4 GiB for Kubernetes and system processes, then makes the rest of the selected instance available to the benchmark. Limits: Only authorized users can enqueue jobs. Jobs run serially, the queue holds 20 active jobs, and each requester may have 3. Query selection is not supported. By default, every query uses 1 warmup and 5 measured iterations for both revisions. |
|
fair to say the benchmarks are fine |
Version 5.0.0 will be released before the dependency upgrade, so keep its guide limited to changes included in that release.
|
this is ready for review |
Why
Prepare DataFusion Distributed for DataFusion 56 and its Arrow 60 / Parquet 60 dependency set. DataFusion is pinned to the exact current
branch-56commit1635ed9.note
The Iceberg dependencies now use Apache-hosted upstream PR commits.
apache/iceberg-rust#3257at0d5eb12supplies the Arrow 60 dependency graph.apache/datafusion-iceberg#99at5398be1supplies the DataFusion 56 API adaptations.Follow-up issue #857 tracks replacing these temporary git pins with crates.io releases once the coordinated DataFusion 56 and Arrow 60-compatible Iceberg releases are published.
What changed
datafusion-clito the exact DataFusion 56 revision, Arrow and Parquet to 60,object_storeto 0.14, and the TPC-H/TPC-DS generators to their Arrow 60-compatible revision.object_storedependency with DataFusion so the AWS feature is enabled on the same 0.14 package.TableProvidermigration in the 6.0 upgrade guide.Commit structure
f9e2855updates Cargo manifests and the lockfile, including the Apache-hosted Iceberg revisions.01aac37aligns the benchmark worker'sobject_storeversion so its AWS client builds with DataFusion 56.86975d1contains the compile and breaking API adaptations.defe313documents downstream upgrade requirements.27074ddcontains the Iceberg and distributed integration expectations.23bfd3econtains planner, TPC-H, sorted TPC-H, and TPC-DS plan snapshots.25a924bmoves the DataFusion 56 migration notes to the 6.0 upgrade guide.The dependency, benchmark-worker, and API commits intentionally keep the dependency graph, benchmark infrastructure fix, and breaking API adaptations independently reviewable.