Repository navigation
Integration of perftest CI with standardised benchmarks - #6514
Merged
Merged
Conversation
|
Can one of the admins verify this patch? |
Coverage summary from CodacySee diff coverage on Codacy
Coverage variation details
Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: Diff coverage details
Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: See your quality gate settings Change summary preferences🚀 Don’t miss a bit, follow what’s new on Codacy. Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more |
hkaiser
added
type: enhancement
category: CI
category: perftests
Contributor
Author
Performance test reportsmall_vector_benchmark
stream_report
foreach_report
transform_reduce_scaling
benchmark_is_heap_until
benchmark_inplace_merge
benchmark_is_heap
benchmark_remove
benchmark_remove_if
minmax_element_performance
small_vector_benchmark
stream_report
foreach_report
transform_reduce_scaling
benchmark_is_heap_until
benchmark_inplace_merge
benchmark_is_heap
benchmark_remove
benchmark_remove_if
small_vector_benchmark
stream_report
minmax_element_performance
stream_report
benchmark_remove_if
|
Contributor
|
@vrnimje Good job! ;) |
vrnimje
force-pushed
the
cdash_perftest_bench
branch
from
July 15, 2024 07:24
839caf7 to
7dd0bf7
Compare
Contributor
|
retest lsu perf |
Collaborator
Performance test reportminmax_element_performance
small_vector_benchmark
future_overhead_report
stream_report
foreach_report
transform_reduce_scaling
benchmark_is_heap_until
benchmark_merge
benchmark_inplace_merge
benchmark_is_heap
benchmark_remove
benchmark_remove_if
|
Collaborator
Performance test report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Collaborator
Performance test report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Coverage summary from CodacySee diff coverage on Codacy
Coverage variation details
Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: Diff coverage details
Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: See your quality gate settings Change summary preferences🚀 Don’t miss a bit, follow what’s new on Codacy. Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more |
Collaborator
Performance test report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Collaborator
Performance test report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
1 similar comment
Collaborator
Performance test report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Collaborator
Performance test report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Collaborator
Performance test report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||
Coverage summary from CodacySee diff coverage on Codacy
Coverage variation details
Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: Diff coverage details
Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: See your quality gate settings Change summary preferencesCodacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more |
Collaborator
Performance test report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Collaborator
Performance test report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Collaborator
Performance test report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Collaborator
Performance test report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Collaborator
Performance test report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Coverage summary from CodacySee diff coverage on Codacy
Coverage variation details
Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: Diff coverage details
Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: See your quality gate settings Change summary preferencesCodacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more |
Collaborator
Performance test report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Collaborator
Performance test report
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Collaborator
Performance test report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Collaborator
Performance test report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Collaborator
Performance test report
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
hkaiser
force-pushed
the
cdash_perftest_bench
branch
2 times, most recently
from
October 21, 2026 13:51
35e18b4 to
048a707
Compare
Contributor
|
@vrnimje In case you're still interested in this work. I think things stabilize now and I plan to merge the overall result soon. Please have a look and comment, if you like. |
Collaborator
|
Commit 048a707cd13484a43fdc06254e533fb7f1b07f5b, Jenkins build 6113 Performance test report
Explanation of SymbolsThe benchmarks ran on medusa08, the results are compared with the references of that node. Percentage Diff. is the change of the median time relative to the reference, positive values mean faster.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Contributor
Author
|
@hkaiser Thanks for following up and improving the implementation. Glad to see things have stabilized. I just scanned the latest changes and it looks good to me. |
hkaiser
force-pushed
the
cdash_perftest_bench
branch
from
October 21, 2026 18:22
048a707 to
fc1b956
Compare
…ences add_hpx_performance_report_test registers a benchmark (enabled with HPX_WITH_TESTS_RUN_BENCHMARKS) as a <name>_perftest target, which runs it a number of times through tools/perftests_ci/driver.py and collects the results; the perftests_plot target compares all results with their references in one process and writes the report (index.html), the plots and the list of measurements that got more than 5% faster or slower; only these are reported, unless HPX_PERFTESTS_REPORT_ALL is set. The timings differ between nodes, so the references are kept per node in tools/perftests_ci/perftest/references/<node>.tar.gz (packed by pack_references.sh); those of medusa08 come from a run on master. A benchmark without a reference is reported and plotted on its own. The configure step lists the registered reports in tools/perftests_ci/perftests.cmake. The other CI jobs don't run the _perftest targets. Signed-off-by: Vedant <[email protected]> Co-authored-by: Hartmut Kaiser <[email protected]> Signed-off-by: Hartmut Kaiser <[email protected]>
The lsu-perftests lane builds and runs all registered performance reports in one ctest run, uploads the results to CDash, and sets a failed status only if a benchmark couldn't be built, run or compared. It comments on the pull request (with the commit, the build, what was measured, a legend, and the plots) only if a measurement got more than 5% faster or slower; the pull request comment "retest lsu perf full" starts a build that reports all measurements. The plots are uploaded to TheHPXProject/hpx-perftest-images together with a README.md that shows the report, and a README.md of the pull request lists the reports of all its builds. The job runs on the first idle medusa node, preferring medusa08 and the nodes that have references, after cancelling the previous jobs of the pull request, and gives up after three hours in the queue. The results of a node without references are uploaded packed next to the plots, ready to become its references. Every build keeps its results packed as references of its node in its artifacts. Waiting for a Slurm job reports the state of the job every ten minutes instead of tracing the wait. Co-authored-by: Vedant <[email protected]> Signed-off-by: Hartmut Kaiser <[email protected]>
Collaborator
|
Commit fc1b956826a5413cabb81ccb759393da48a85bf8, Jenkins build 6114 Performance test report
Explanation of SymbolsThe benchmarks ran on medusa08, the results are compared with the references of that node. Percentage Diff. is the change of the median time relative to the reference, positive values mean faster.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
hkaiser
force-pushed
the
cdash_perftest_bench
branch
from
October 21, 2026 18:46
fc1b956 to
d954dad
Compare
Up to standards ✅🟢 Issues
|
Collaborator
|
Commit d954dad605bd0ca4f96ada402469bd61e82152cf, Jenkins build 6115 Performance test report
Explanation of SymbolsThe benchmarks ran on medusa08, the results are compared with the references of that node. Percentage Diff. is the change of the median time relative to the reference, positive values mean faster.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Contributor
|
retest lsu perf full |
Collaborator
|
Commit d954dad605bd0ca4f96ada402469bd61e82152cf, Jenkins build 6116, full report (requested) Performance test report
Explanation of SymbolsThe benchmarks ran on medusa08, the results are compared with the references of that node. Percentage Diff. is the change of the median time relative to the reference, positive values mean faster.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
…tions
The comparison bootstrapped the median of all timings of all executions of a
benchmark as if they were independent. The timings of one execution depend on
each other, and most of the variation is between the executions, so the
confidence interval came out far too narrow, and the noise between executions
showed up as changes of 5 to 10 percent (e.g. the stream Copy benchmark with
the scheduler_executor). The perftest driver now records how many timings each
execution added ("executions"), and the comparison bootstraps the medians of
the executions. Replaying earlier CI runs of the same code against their
reference doesn't classify any measurement as more than 5 percent faster or
slower any more; the unstable measurements are classified as uncertain
instead.
The medusa08 references get the number of timings per execution as well, and
the benchmarks run 30 times instead of 15 by default, which makes the
interval of the medians narrower.
Signed-off-by: Hartmut Kaiser <[email protected]>
1 of 3 tasks
Contributor
|
retest lsu perf |
1 similar comment
Contributor
|
retest lsu perf |
A job submitted for a busy node waits for that node, even if others become idle in the meantime: when no node was idle at submission, the performance tests waited for medusa08 while other medusa nodes were free. entry.sh now waits for one of the candidate nodes to be idle before submitting, out of the same 3 hour budget as the queue, and the job fails without submitting if none becomes idle in time. The nodes with references are still preferred. The performance tests now run in the medusa partition, which has all medusa nodes; jenkins-compute has only medusa02 to medusa10, which the build matrix keeps busy. Nodes that aren't in the partition are skipped, a job asking for them would never start. All scripts log what they do and why, as "[perftests <time> <script>] ..." lines that stand out in the traced output: the build, the nodes with references, the idle nodes whenever they change, the chosen node, the Slurm and ctest status, every benchmark that runs, why the pull request gets a comment or not, the uploads and the GitHub status. Signed-off-by: Hartmut Kaiser <[email protected]>
hkaiser
force-pushed
the
cdash_perftest_bench
branch
from
October 22, 2026 16:30
8188106 to
2b55401
Compare
Collaborator
|
Commit 2b5540148156494a9d19a70fef9f149725cf9ba0, Jenkins build 6127 medusa00 has no references for some of the benchmarks yet. Its references, completed with the results of this run, are uploaded as medusa00.tar.gz, copy it to tools/perftests_ci/perftest/references to use them. Performance test report
Explanation of SymbolsThe benchmarks ran on medusa00, the results are compared with the references of that node. Percentage Diff. is the change of the median time relative to the reference, positive values mean faster.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
To follow the performance of HPX over time, the performance tests of every master commit that is built are now added to the performance history in TheHPXProject/hpx-performance. These runs always run on the same node, medusa00 (configuration_slurm_history_node), so that they can be compared with each other; it is not in jenkins-compute, so the build matrix never runs on it. They wait up to a day for it and report all measurements. batch.sh records the run with tools/perftests_ci/perftests_record.py: the commit (hash, parents, dates, pull request, position on the first parent history), when and where the benchmarks ran, the Jenkins build, the configuration and the status (metadata.json), and the medians of all measurements, overall and per execution (summary.json). entry.sh uploads it with upload_history.sh, also if some of the benchmarks failed; a failed upload turns the status red. Each run gets its own directory, <year>/<node>/<commit date>-<commit>-b<build>, with its report, the record and the results of all benchmarks as measured; the plots of the comparison are left out, perftests_plot.py can draw them again. runs.tsv lists all runs and timeline.csv the medians of all measurements of all runs. With every run, tools/perftests_ci/perftests_timeline.py draws a graph per benchmark over time (SVG, so that the repository stores only their differences), which the README of the repository shows. tools/perftests_ci/perftests_history.py rebuilds timeline.csv from the run directories, optionally for one node, one benchmark or a range of dates. The commits are made as the STE||AR-Bot, and the first run sets up the empty repository with the Boost license. Signed-off-by: Hartmut Kaiser <[email protected]>
A benchmark that is new, or whose measurements changed, has no reference on the node the performance tests ran on. All references of that node are now made again from the run, so that they are measured together, instead of adding only the results of the new benchmark to the existing references. A benchmark whose measurements differ from its reference counts as one without a reference instead of failing the comparison. The new references are uploaded to TheHPXProject/hpx-perftest-images in any case: also if the pull request gets no comment, and for builds of a branch (as branch/<branch>/<build>). A failed upload turns the status red. Signed-off-by: Hartmut Kaiser <[email protected]>
The results of all benchmarks of Jenkins build 6127 on medusa00, as uploaded to TheHPXProject/hpx-perftest-images (pr/6514/6127/references/medusa00.tar.gz), so that pull requests that run on medusa00 are compared with them. Signed-off-by: Hartmut Kaiser <[email protected]>
hkaiser
force-pushed
the
cdash_perftest_bench
branch
from
October 22, 2026 20:49
78f2d3e to
14cef03
Compare
This was referenced Oct 22, 2026
hkaiser
added a commit
that referenced
this pull request
Oct 22, 2026
Restore the release notes update that was lost when release-2.0.X was rebuilt on master eaff5b9, and describe the PRs merged since the previous rebuild (#6514, #7767, #7816, #7817, #7819, #7821, #7824, #7831). Regenerate the closed issue and pull request lists for release-2.0.X at 4cb902e against the v1.11.0 tag (775 PRs, 167 issues). Signed-off-by: Hartmut Kaiser <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed Changes
Any background context you want to provide?
Extension to #6448
Sample run on CDash: https://my.cdash.org/builds/2589647