Integration of perftest CI with standardised benchmarks #6514

vrnimje · 2024-06-21T07:28:50Z

Proposed Changes

Improved the perftest CI with CDash integration
Added separate cmake function for adding performance tests with specific options for benchmarking purposes, which also creates a custom target for generating the plots

Any background context you want to provide?

Extension to #6448
Sample run on CDash: https://my.cdash.org/builds/2589647

jenkins-cscs · 2024-06-21T07:29:02Z

Can one of the admins verify this patch?

codacy-production · 2024-06-21T11:15:41Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ -0.06%	✅ 81.75%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`17bbc63`)	222489	189245	85.06%
Head commit (`6072f1b`)	196545 (-25944)	167059 (-22186)	85.00% (-0.06%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#6514)	263	215	81.75%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{🚀 Don’t miss a bit, follow what’s new on Codacy.}

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

vrnimje · 2024-07-11T05:58:32Z

Performance test report

small_vector_benchmark

hpx::small_vector, <fxn<void()>, 2>: Performance is better by 0.46 % (KS-stat: 0.9583333333333334)

stream_report

Stream benchmark - Scale, fork_join_executor_default_fork_join_policy_allocator: Performance is better by 12.69 % (KS-stat: 0.9583333333333334)
Stream benchmark - Triad, fork_join_executor_default_fork_join_policy_allocator: Performance is better by 3.25 % (KS-stat: 0.875)
Stream benchmark - Scale, scheduler_executor_default_scheduler_executor_allocator: Performance is better by 18.40 % (KS-stat: 1.0)
Stream benchmark - Add, scheduler_executor_default_scheduler_executor_allocator: Performance is better by 10.84 % (KS-stat: 0.7916666666666666)

foreach_report

for_each, fork_join_executor: Performance is better by 6.67 % (KS-stat: 0.9166666666666666)

transform_reduce_scaling

transform_reduce, par: Performance is better by 57.55 % (KS-stat: 1.0)
transform_reduce_old, par: Performance is better by 69.66 % (KS-stat: 1.0)

benchmark_is_heap_until

hpx::heap_until, seq: Performance is better by 35.65 % (KS-stat: 1.0)
hpx::heap_until, par: Performance is better by 32.25 % (KS-stat: 0.9583333333333334)

benchmark_inplace_merge

hpx::inplace_merge, seq: Performance is better by 41.21 % (KS-stat: 0.9166666666666666)
hpx::inplace_merge, par_unseq: Performance is better by 49.60 % (KS-stat: 1.0)

benchmark_is_heap

hpx::is_heap, seq: Performance is better by 100.00 % (KS-stat: 1.0)

benchmark_remove

hpx::remove, seq: Performance is better by 21.37 % (KS-stat: 1.0)
hpx::remove, par: Performance is better by 30.38 % (KS-stat: 0.9583333333333334)
hpx::remove, par_unseq: Performance is better by 31.96 % (KS-stat: 0.9166666666666666)

benchmark_remove_if

hpx::remove_if, seq: Performance is better by 32.02 % (KS-stat: 1.0)
hpx::remove_if, par: Performance is better by 38.20 % (KS-stat: 1.0)
hpx::remove_if, par_unseq: Performance is better by 37.82 % (KS-stat: 1.0)

minmax_element_performance

hpx::min, par: Performance is better by 19.53 % (KS-stat: 0.9166666666666666)

small_vector_benchmark

hpx::small_vector, <fxn<void()>, 1>: Performance is worse by 9.37 % (KS-stat: 0.7916666666666666)
hpx::small_vector, <fxn<void()>, 8>: Performance is better by 1.68 % (KS-stat: 0.9166666666666666)

stream_report

stream benchmark - Copy, fork_join_executor_default_fork_join_policy_allocator: Performance is better by 6.28 % (KS-stat: 1.0)
Stream benchmark - Scale, fork_join_executor_default_fork_join_policy_allocator: Performance is better by 13.83 % (KS-stat: 1.0)
Stream benchmark - Add, fork_join_executor_default_fork_join_policy_allocator: Performance is better by 4.41 % (KS-stat: 0.7916666666666666)
Stream benchmark - Triad, fork_join_executor_default_fork_join_policy_allocator: Performance is better by 2.72 % (KS-stat: 0.75)
Stream benchmark - Scale, scheduler_executor_default_scheduler_executor_allocator: Performance is better by 18.22 % (KS-stat: 0.9583333333333334)

foreach_report

for_each, fork_join_executor: Performance is better by 6.40 % (KS-stat: 0.9166666666666666)

transform_reduce_scaling

transform_reduce, par: Performance is better by 57.40 % (KS-stat: 1.0)
transform_reduce_old, par: Performance is better by 69.81 % (KS-stat: 1.0)

benchmark_is_heap_until

hpx::heap_until, seq: Performance is better by 36.00 % (KS-stat: 1.0)
hpx::heap_until, par: Performance is better by 32.02 % (KS-stat: 0.9583333333333334)

benchmark_inplace_merge

hpx::inplace_merge, seq: Performance is better by 41.25 % (KS-stat: 0.9583333333333334)
hpx::inplace_merge, par_unseq: Performance is better by 49.76 % (KS-stat: 1.0)

benchmark_is_heap

hpx::is_heap, seq: Performance is better by 100.00 % (KS-stat: 1.0)

benchmark_remove

hpx::remove, seq: Performance is better by 21.60 % (KS-stat: 0.9583333333333334)
hpx::remove, par: Performance is better by 31.05 % (KS-stat: 1.0)
hpx::remove, par_unseq: Performance is better by 33.81 % (KS-stat: 1.0)

benchmark_remove_if

hpx::remove_if, seq: Performance is better by 29.03 % (KS-stat: 1.0)
hpx::remove_if, par: Performance is better by 37.92 % (KS-stat: 1.0)
hpx::remove_if, par_unseq: Performance is better by 37.52 % (KS-stat: 1.0)

small_vector_benchmark

hpx::small_vector, <int, 1>: Performance is worse by 2.90 % (KS-stat: 0.7916666666666666)
hpx::small_vector, <fxn<void()>, 1>: Performance is better by 8.86 % (KS-stat: 0.7916666666666666)

stream_report

stream benchmark - Copy, fork_join_executor_default_fork_join_policy_allocator: Performance is better by 1.72 % (KS-stat: 0.7916666666666666)

minmax_element_performance

hpx::min, par: Performance is worse by 89.74 % (KS-stat: 0.75)

stream_report

stream benchmark - Copy, fork_join_executor_default_fork_join_policy_allocator: Performance is worse by 3.32 % (KS-stat: 1.0)
Stream benchmark - Scale, fork_join_executor_default_fork_join_policy_allocator: Performance is worse by 21.33 % (KS-stat: 0.8333333333333334)
Stream benchmark - Add, fork_join_executor_default_fork_join_policy_allocator: Performance is worse by 16.76 % (KS-stat: 0.7916666666666666)
Stream benchmark - Triad, fork_join_executor_default_fork_join_policy_allocator: Performance is worse by 3.95 % (KS-stat: 0.75)
Stream benchmark - Add, scheduler_executor_default_scheduler_executor_allocator: Performance is better by 15.89 % (KS-stat: 0.9166666666666666)

benchmark_remove_if

hpx::remove_if, seq: Performance is better by 4.40 % (KS-stat: 0.75)

Pansysk75 · 2024-07-11T10:53:56Z

@vrnimje Good job! ;)

Signed-off-by: Vedant <[email protected]>

…x perf test Signed-off-by: Vedant <[email protected]>

Signed-off-by: Vedant <[email protected]>

…minor changes to plotting script * Further simplified the default template for the benchmark Signed-off-by: Vedant <[email protected]>

…accordingly Signed-off-by: Vedant <[email protected]>

…orrect option Signed-off-by: Vedant <[email protected]>

Signed-off-by: Vedant <[email protected]>

…, and made minor changes to perftests ci Signed-off-by: Vedant <[email protected]>

Signed-off-by: Vedant <[email protected]>

…elines, and fixed clang- and cmake-formatting issues Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-15T07:16:49Z

Performance test report

Test name	Executor	Mean Percentage Diff.	Coefficient of variation
minmax_element_performance
hpx::minmax	par	1.96 %	1.18651
hpx::min	par	-0.11 %	15.25031
hpx::max	par	0.05 %	42.50429
small_vector_benchmark
hpx::small_vector	<int, 8>	0.52 %	0.47845
future_overhead_report
future overhead - create_thread_hierarchical - latch	no-executor	0.20 %	14.95532
stream_report
Stream benchmark - Add	scheduler_executor_default_scheduler_executor_allocator	1.57 %	0.38104
transform_reduce_scaling
transform_reduce	par	-0.05 %	15.06714
benchmark_is_heap_until
hpx::heap_until	par	-0.28 %	6.53407
benchmark_merge
hpx::merge	seq	1.25 %	0.04249
hpx::merge	par	-5.67 %	0.12528
hpx::merge	par_unseq	-1.26 %	0.45807
benchmark_inplace_merge
hpx::inplace_merge	par	10.49 %	0.15894
hpx::inplace_merge	par_unseq	11.33 %	0.12265
benchmark_is_heap
hpx::is_heap	par	0.41 %	7.33730
benchmark_remove_if
hpx::remove_if	seq	-0.90 %	0.11926
benchmark_partial_sort
hpx::partial_sort, size: 1000, step: 1	seq	5.45 %	0.02869
hpx::partial_sort, size: 100000	seq	-1.46 %	0.09224
benchmark_partial_sort_parallel
hpx::partial_sort, size: 100000	par	9.37 %	0.02204
hpx::partial_sort, size: 100000, step: 1000	par	9.83 %	0.05209
benchmark_nth_element_parallel
hpx::nth_element, size: 1000, step: 1	par	-9.17 %	0.98629

…n CDash, and added them to the build artifacts Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-15T10:11:56Z

Performance test report

Test name	Executor	Mean Percentage Diff.	Coefficient of variation
minmax_element_performance
hpx::minmax	par	-1.19 %	1.76652
hpx::min	par	-0.42 %	4.10981
hpx::max	par	-0.03 %	21.59520
small_vector_benchmark
hpx::small_vector	<fxn<void()>, 1>	-0.70 %	0.25417
future_overhead_report
future overhead - create_thread_hierarchical - latch	no-executor	-12.63 %	0.30442
stream_report
stream benchmark - Copy	parallel_executor_default_parallel_policy_allocator	-4.30 %	0.19204
Stream benchmark - Scale	parallel_executor_default_parallel_policy_allocator	-2.52 %	0.44144
Stream benchmark - Scale	fork_join_executor_default_fork_join_policy_allocator	1.32 %	0.03324
stream benchmark - Copy	scheduler_executor_default_scheduler_executor_allocator	-2.47 %	0.45755
benchmark_is_heap_until
hpx::heap_until	par	0.00 %	545.61520
benchmark_merge
hpx::merge	seq	1.42 %	0.04377
hpx::merge	par	-3.80 %	0.16191
hpx::merge	par_unseq	-1.84 %	0.34002
benchmark_inplace_merge
hpx::inplace_merge	seq	-1.15 %	0.06430
hpx::inplace_merge	par	9.01 %	0.23894
hpx::inplace_merge	par_unseq	10.28 %	0.12878
benchmark_is_heap
hpx::is_heap	par	-1.09 %	2.25798
benchmark_remove_if
hpx::remove_if	seq	0.90 %	0.12822
hpx::remove_if	par	1.14 %	0.07750
hpx::remove_if	par_unseq	1.05 %	0.26647
benchmark_partial_sort
hpx::partial_sort, size: 1000, step: 1	seq	11.04 %	0.01236
hpx::partial_sort, size: 100000	seq	-1.33 %	0.11409
benchmark_partial_sort_parallel
hpx::partial_sort, size: 1000, step: 1	par	1.46 %	0.10163
hpx::partial_sort, size: 100000, step: 1000	par	11.22 %	0.04772
benchmark_nth_element_parallel
hpx::nth_element, size: 100000, step: 5000	par	-0.83 %	1.46625

Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-15T11:12:53Z

Performance test report

Test name	Executor	Mean Percentage Diff.	Coefficient of variation
minmax_element_performance
hpx::minmax	par	1.44 %	1.37947
hpx::min	par	0.61 %	2.33319
hpx::max	par	-1.19 %	1.02057
small_vector_benchmark
hpx::small_vector	<int, 8>	0.53 %	0.48545
future_overhead_report
future overhead - create_thread_hierarchical - latch	no-executor	-0.57 %	6.01747
stream_report
Stream benchmark - Scale	parallel_executor_default_parallel_policy_allocator	0.10 %	15.05362
Stream benchmark - Add	parallel_executor_default_parallel_policy_allocator	0.99 %	0.49465
Stream benchmark - Scale	scheduler_executor_default_scheduler_executor_allocator	-3.21 %	0.41208
foreach_report
for_each	scheduler_executor	1.68 %	0.45645
transform_reduce_scaling
transform_reduce	par	-0.35 %	3.86105
benchmark_is_heap_until
hpx::heap_until	par	0.24 %	8.86754
benchmark_merge
hpx::merge	seq	1.31 %	0.04086
hpx::merge	par	-4.19 %	0.16176
benchmark_inplace_merge
hpx::inplace_merge	par	3.85 %	0.38700
hpx::inplace_merge	par_unseq	2.51 %	0.30816
benchmark_is_heap
hpx::is_heap	par	0.46 %	6.13407
benchmark_partial_sort
hpx::partial_sort, size: 1000, step: 1	seq	2.12 %	0.08383
hpx::partial_sort, size: 100000	seq	-1.61 %	0.08373
benchmark_partial_sort_parallel
hpx::partial_sort, size: 1000, step: 1	par	4.87 %	0.03424
hpx::partial_sort, size: 100000	par	6.46 %	0.04234
hpx::partial_sort, size: 100000, step: 1000	par	8.15 %	0.06312
benchmark_nth_element_parallel
hpx::nth_element, size: 1000, step: 1	par	-11.40 %	0.95884
hpx::nth_element, size: 100000, step: 5000	par	-0.85 %	1.66926

Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-15T14:01:31Z

Performance test report

Test name	Executor	Mean Percentage Diff.	Coefficient of variation
minmax_element_performance
hpx::minmax	par	-0.02 %	38.85135
hpx::min	par	-0.45 %	3.00622
hpx::max	par	-0.79 %	1.32714
future_overhead_report
future overhead - create_thread_hierarchical - latch	no-executor	2.13 %	1.30068
stream_report
stream benchmark - Copy	parallel_executor_default_parallel_policy_allocator	-0.50 %	1.70264
Stream benchmark - Add	parallel_executor_default_parallel_policy_allocator	-1.72 %	0.36338
Stream benchmark - Triad	parallel_executor_default_parallel_policy_allocator	1.15 %	0.43536
stream benchmark - Copy	scheduler_executor_default_scheduler_executor_allocator	-3.75 %	0.23468
benchmark_is_heap_until
hpx::heap_until	par	-0.32 %	6.67886
hpx::heap_until	par_unseq	0.55 %	1.61591
benchmark_merge
hpx::merge	seq	1.39 %	0.03691
hpx::merge	par	-4.63 %	0.15019
hpx::merge	par_unseq	-2.20 %	0.29576
benchmark_inplace_merge
hpx::inplace_merge	par	0.78 %	2.12736
hpx::inplace_merge	par_unseq	-0.17 %	7.34590
benchmark_is_heap
hpx::is_heap	par	1.70 %	1.47054
benchmark_remove_if
hpx::remove_if	seq	0.89 %	0.11261
hpx::remove_if	par	0.98 %	0.08889
hpx::remove_if	par_unseq	1.04 %	0.26621
benchmark_partial_sort
hpx::partial_sort, size: 1000, step: 1	seq	7.16 %	0.02518
hpx::partial_sort, size: 100000	seq	-0.89 %	0.14436
benchmark_partial_sort_parallel
hpx::partial_sort, size: 100000	par	2.15 %	0.12404
hpx::partial_sort, size: 100000, step: 1000	par	4.85 %	0.09645
benchmark_nth_element_parallel
hpx::nth_element, size: 100000, step: 5000	par	-0.98 %	1.18164

…s again Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-15T14:38:12Z

Performance test report

Test name	Executor	Mean Percentage Diff.	Coefficient of variation
minmax_element_performance
hpx::minmax	par	0.76 %	2.77272
hpx::min	par	0.79 %	1.61304
hpx::max	par	-0.17 %	5.72621
future_overhead_report
future overhead - create_thread_hierarchical - latch	no-executor	-0.03 %	22.37570
stream_report
stream benchmark - Copy	parallel_executor_default_parallel_policy_allocator	-0.49 %	1.92481
Stream benchmark - Scale	parallel_executor_default_parallel_policy_allocator	4.08 %	0.24543
Stream benchmark - Scale	scheduler_executor_default_scheduler_executor_allocator	-0.25 %	4.56995
Stream benchmark - Add	scheduler_executor_default_scheduler_executor_allocator	1.58 %	0.32848
foreach_report
for_each	scheduler_executor	0.34 %	2.50820
transform_reduce_scaling
transform_reduce_old	par	1.95 %	0.24662
benchmark_is_heap_until
hpx::heap_until	par	-0.40 %	4.96255
hpx::heap_until	par_unseq	1.71 %	0.42793
benchmark_merge
hpx::merge	seq	1.00 %	0.05525
hpx::merge	par	-6.01 %	0.11866
hpx::merge	par_unseq	-2.66 %	0.22683
benchmark_inplace_merge
hpx::inplace_merge	par	-0.25 %	5.96318
hpx::inplace_merge	par_unseq	3.69 %	0.24123
benchmark_is_heap
hpx::is_heap	par	0.24 %	12.59392
hpx::is_heap	par_unseq	-1.71 %	0.44053
benchmark_remove
hpx::remove	seq	2.08 %	0.08116
hpx::remove	par	1.18 %	0.11226
hpx::remove	par_unseq	0.90 %	0.09645
benchmark_remove_if
hpx::remove_if	seq	1.00 %	0.10714
hpx::remove_if	par	1.29 %	0.07869
hpx::remove_if	par_unseq	1.32 %	0.18357
benchmark_partial_sort
hpx::partial_sort, size: 1000, step: 1	seq	9.64 %	0.01445
hpx::partial_sort, size: 100000	seq	-1.13 %	0.09270
benchmark_partial_sort_parallel
hpx::partial_sort, size: 1000, step: 1	par	1.62 %	0.10247
hpx::partial_sort, size: 100000, step: 1000	par	10.23 %	0.05292
benchmark_nth_element_parallel
hpx::nth_element, size: 100000, step: 5000	par	0.70 %	0.39393

codacy-production · 2024-08-15T18:01:18Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ -0.06%	✅ 86.70%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`17bbc63`)	222489	189245	85.06%
Head commit (`ff62d46`)	196730 (-25759)	167219 (-22026)	85.00% (-0.06%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#6514)	436	378	86.70%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-17T07:13:02Z

Performance test report

Test name	Executor	Mean Percentage Diff.	Coefficient of variation
minmax_element_performance
hpx::minmax	par	4.10 %	0.33835
hpx::min	par	2.22 %	0.47294
future_overhead_report
future overhead - create_thread_hierarchical - latch	no-executor	-2.48 %	1.14423
stream_report
Stream benchmark - Scale	parallel_executor_default_parallel_policy_allocator	5.08 %	0.18510
stream benchmark - Copy	scheduler_executor_default_scheduler_executor_allocator	-5.06 %	0.41326
Stream benchmark - Scale	scheduler_executor_default_scheduler_executor_allocator	2.40 %	0.24650
Stream benchmark - Add	scheduler_executor_default_scheduler_executor_allocator	1.53 %	0.32002
transform_reduce_scaling
transform_reduce	par	0.72 %	1.15210
transform_reduce_old	par	1.19 %	0.33932
benchmark_is_heap_until
hpx::heap_until	par	5.66 %	0.26522
benchmark_merge
hpx::merge	seq	1.49 %	0.02982
hpx::merge	par	-0.84 %	0.47515
hpx::merge	par_unseq	-1.12 %	0.38487
benchmark_inplace_merge
hpx::inplace_merge	par	8.98 %	0.16203
hpx::inplace_merge	par_unseq	10.69 %	0.08418
benchmark_is_heap
hpx::is_heap	par	5.14 %	0.34268
benchmark_remove
hpx::remove	seq	1.01 %	0.11027
benchmark_remove_if
hpx::remove_if	seq	0.89 %	0.10283
benchmark_partial_sort
hpx::partial_sort, size: 1000, step: 1	seq	4.11 %	0.02851
hpx::partial_sort, size: 100000	seq	-1.11 %	0.09047
benchmark_partial_sort_parallel
hpx::partial_sort, size: 1000, step: 1	par	-5.42 %	0.02644
hpx::partial_sort, size: 100000	par	5.34 %	0.04335
hpx::partial_sort, size: 100000, step: 1000	par	7.37 %	0.06949

… image name for CDash Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-17T09:33:13Z

Performance test report

Test name	Executor	Mean Percentage Diff.	Coefficient of variation
minmax_element_performance
hpx::min	par	-1.25 %	0.45074
future_overhead_report
future overhead - create_thread_hierarchical - latch	no-executor	9.20 %	0.25129
stream_report
Stream benchmark - Add	parallel_executor_default_parallel_policy_allocator	1.03 %	0.30720
Stream benchmark - Triad	parallel_executor_default_parallel_policy_allocator	0.91 %	0.36907
stream benchmark - Copy	scheduler_executor_default_scheduler_executor_allocator	3.72 %	0.78148
Stream benchmark - Triad	scheduler_executor_default_scheduler_executor_allocator	1.53 %	0.22015
benchmark_merge
hpx::merge	par	0.55 %	0.47600
benchmark_inplace_merge
hpx::inplace_merge	par	0.34 %	3.88061
hpx::inplace_merge	par_unseq	1.05 %	1.06177
benchmark_is_heap
hpx::is_heap	par	0.56 %	1.41438
benchmark_remove
hpx::remove	par_unseq	-0.72 %	1.45818
benchmark_remove_if
hpx::remove_if	par_unseq	0.10 %	10.37297
benchmark_partial_sort
hpx::partial_sort, size: 1000, step: 1	seq	1.52 %	0.07678
benchmark_partial_sort_parallel
hpx::partial_sort, size: 1000, step: 1	par	6.47 %	0.04699
hpx::partial_sort, size: 100000	par	2.04 %	0.11232
benchmark_nth_element_parallel
hpx::nth_element, size: 1000, step: 1	par	-6.31 %	0.93441
hpx::nth_element, size: 100000, step: 5000	par	0.23 %	5.23035

… to use medians Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-19T16:28:36Z

Performance test report

Test name	Executor	Mean Percentage Diff.	Coefficient of variation
minmax_element_performance
hpx::max	par	0.75 %	0.71582
small_vector_benchmark
hpx::small_vector	<int, 2>	-1.09 %	0.43155
hpx::small_vector	<int, 4>	-1.00 %	0.21861
hpx::small_vector	<int, 16>	1.39 %	0.00417
hpx::small_vector	<fxn<void()>, 1>	2.63 %	0.03075
hpx::small_vector	<fxn<void()>, 2>	1.55 %	0.05928
hpx::small_vector	<fxn<void()>, 4>	4.30 %	0.00848
hpx::small_vector	<fxn<void()>, 8>	1.35 %	0.04220
hpx::small_vector	<fxn<void()>, 16>	3.77 %	0.00194
future_overhead_report
future overhead - create_thread_hierarchical - latch	no-executor	7.44 %	0.25946
stream_report
stream benchmark - Copy	parallel_executor_default_parallel_policy_allocator	-1.26 %	0.45600
Stream benchmark - Add	parallel_executor_default_parallel_policy_allocator	0.53 %	0.20760
Stream benchmark - Triad	parallel_executor_default_parallel_policy_allocator	0.67 %	0.19165
stream benchmark - Copy	scheduler_executor_default_scheduler_executor_allocator	2.15 %	0.16984
Stream benchmark - Add	scheduler_executor_default_scheduler_executor_allocator	1.14 %	0.74768
Stream benchmark - Triad	scheduler_executor_default_scheduler_executor_allocator	0.77 %	0.24945
transform_reduce_scaling
transform_reduce	par	-0.52 %	0.43732
transform_reduce_old	par	-0.67 %	0.32381
benchmark_is_heap_until
hpx::heap_until	seq	16.66 %	0.00071
benchmark_merge
hpx::merge	par	1.20 %	0.24594
hpx::merge	par_unseq	1.23 %	0.30648
benchmark_inplace_merge
hpx::inplace_merge	seq	0.39 %	0.16341
hpx::inplace_merge	par	-7.56 %	0.08207
hpx::inplace_merge	par_unseq	-9.89 %	0.05748
benchmark_is_heap
hpx::is_heap	seq	16.64 %	0.00047
hpx::is_heap	par	-0.02 %	3.44765
benchmark_remove
hpx::remove	par_unseq	-1.22 %	0.02323
benchmark_partial_sort
hpx::partial_sort, size: 1000, step: 1	seq	5.18 %	0.01505
benchmark_partial_sort_parallel
hpx::partial_sort, size: 1000, step: 1	par	7.42 %	0.01615
hpx::partial_sort, size: 100000	par	3.74 %	0.06045
hpx::partial_sort, size: 100000, step: 1000	par	-2.06 %	0.02687

…ce cmdline options with hpx: Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-22T11:13:35Z

Performance test report

Test name	Executor	Percentage Diff.	Classification
future_overhead_report
future overhead - create_thread_hierarchical - latch	no-executor	1.57 %	?
stream_report
Stream benchmark - Triad	parallel_executor_default_parallel_policy_allocator	2.32 %	-
stream benchmark - Copy	scheduler_executor_default_scheduler_executor_allocator	5.63 %	--
Stream benchmark - Scale	scheduler_executor_default_scheduler_executor_allocator	1.54 %	-
Stream benchmark - Add	scheduler_executor_default_scheduler_executor_allocator	1.31 %	-
Stream benchmark - Triad	scheduler_executor_default_scheduler_executor_allocator	1.89 %	-
benchmark_is_heap_until
hpx::heap_until	seq	16.65 %	---
benchmark_inplace_merge
hpx::inplace_merge	par	0.77 %	?
hpx::inplace_merge	par_unseq	-1.58 %	+
benchmark_is_heap
hpx::is_heap	seq	16.66 %	---
hpx::is_heap	par	0.21 %	?
benchmark_partial_sort_parallel
hpx::partial_sort, size: 100000	par	3.07 %	-
hpx::partial_sort, size: 100000, step: 1000	par	-3.09 %	+
benchmark_nth_element_parallel
hpx::nth_element, size: 1000, step: 1	par	-15.38 %	+++
hpx::nth_element, size: 100000, step: 5000	par	-7.18 %	++

Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-22T14:03:17Z

Performance test report

Test name	Executor	Percentage Diff.	Classification
minmax_element_performance
hpx::minmax	par	-0.74 %	?
hpx::max	par	-1.84 %	+
small_vector_benchmark
hpx::small_vector	int, 1	2.99 %	-
hpx::small_vector	int, 2	-1.51 %	+
hpx::small_vector	int, 4	2.69 %	-
hpx::small_vector	int, 8	15.83 %	---
hpx::small_vector	int, 16	35.71 %	---
hpx::small_vector	fxn:void(), 1	7.22 %	--
hpx::small_vector	fxn:void(), 2	6.00 %	--
hpx::small_vector	fxn:void(), 4	5.38 %	--
hpx::small_vector	fxn:void(), 8	8.88 %	--
future_overhead_report
future overhead - create_thread_hierarchical - latch	no-executor	1.51 %	?
stream_report
Stream benchmark - Add	parallel_executor_default_parallel_policy_allocator	1.55 %	-
Stream benchmark - Triad	parallel_executor_default_parallel_policy_allocator	2.98 %	-
stream benchmark - Copy	scheduler_executor_default_scheduler_executor_allocator	6.43 %	--
Stream benchmark - Scale	scheduler_executor_default_scheduler_executor_allocator	1.30 %	?
Stream benchmark - Add	scheduler_executor_default_scheduler_executor_allocator	2.28 %	-
Stream benchmark - Triad	scheduler_executor_default_scheduler_executor_allocator	2.07 %	-
foreach_report
for_each	parallel_executor	-0.02 %	(+)
benchmark_is_heap_until
hpx::heap_until	seq	16.65 %	---
benchmark_merge
hpx::merge	par	-3.96 %	+
benchmark_inplace_merge
hpx::inplace_merge	par	1.18 %	?
hpx::inplace_merge	par_unseq	-1.16 %	+
benchmark_is_heap
hpx::is_heap	seq	16.65 %	---
hpx::is_heap	par	-3.49 %	++
benchmark_partial_sort_parallel
hpx::partial_sort, size: 100000	par	2.79 %	-
hpx::partial_sort, size: 100000, step: 1000	par	2.29 %	-
benchmark_nth_element_parallel
hpx::nth_element, size: 1000, step: 1	par	-17.33 %	+++
hpx::nth_element, size: 100000, step: 5000	par	-8.50 %	++

…ternal benchmarking iterations to 1000 Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-23T10:18:04Z

Performance test report

Test name	Executor	Percentage Diff.	Classification
minmax_element_performance
hpx::minmax	par	1.64 %	-
small_vector_benchmark
hpx::small_vector	int, 1	2.53 %	-
hpx::small_vector	int, 2	-3.50 %	+
hpx::small_vector	int, 8	15.97 %	---
hpx::small_vector	int, 16	34.28 %	---
hpx::small_vector	fxn:void(), 1	6.99 %	--
hpx::small_vector	fxn:void(), 2	3.54 %	-
hpx::small_vector	fxn:void(), 4	5.39 %	--
hpx::small_vector	fxn:void(), 8	8.94 %	--
future_overhead_report
future overhead - create_thread_hierarchical - latch	no-executor	-1.82 %	+
stream_report
stream benchmark - Copy	parallel_executor_default_parallel_policy_allocator	2.18 %	-
Stream benchmark - Add	parallel_executor_default_parallel_policy_allocator	2.77 %	-
Stream benchmark - Triad	parallel_executor_default_parallel_policy_allocator	4.53 %	--
stream benchmark - Copy	scheduler_executor_default_scheduler_executor_allocator	6.86 %	--
Stream benchmark - Scale	scheduler_executor_default_scheduler_executor_allocator	2.38 %	-
Stream benchmark - Add	scheduler_executor_default_scheduler_executor_allocator	1.77 %	-
Stream benchmark - Triad	scheduler_executor_default_scheduler_executor_allocator	3.74 %	--
benchmark_is_heap_until
hpx::heap_until	seq	16.71 %	---
benchmark_merge
hpx::merge	par_unseq	-2.83 %	+
benchmark_inplace_merge
hpx::inplace_merge	par	0.93 %	?
hpx::inplace_merge	par_unseq	-1.24 %	+
benchmark_is_heap
hpx::is_heap	seq	16.71 %	---
hpx::is_heap	par	0.48 %	?
benchmark_partial_sort
hpx::partial_sort, size: 1000, step: 1	seq	10.67 %	---
benchmark_partial_sort_parallel
hpx::partial_sort, size: 1000, step: 1	par	3.01 %	-
hpx::partial_sort, size: 100000	par	1.75 %	-
hpx::partial_sort, size: 100000, step: 1000	par	5.22 %	--
benchmark_nth_element_parallel
hpx::nth_element, size: 1000, step: 1	par	-16.44 %	+++
hpx::nth_element, size: 100000, step: 5000	par	-7.93 %	++

…rftest_plot script Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-24T11:38:41Z

Performance test report

Test name	Executor	Percentage Diff.	Classification
small_vector_benchmark
hpx::small_vector	fxn:void(), 1	-2.20 %	+
hpx::small_vector	fxn:void(), 4	-4.90 %	+
hpx::small_vector	fxn:void(), 8	-4.02 %	+
hpx::small_vector	fxn:void(), 16	-3.08 %	+
stream_report
Stream benchmark - Triad	parallel_executor_default_parallel_policy_allocator	-1.86 %	+
benchmark_is_heap_until
hpx::heap_until	par	11.95 %	---
benchmark_inplace_merge
hpx::inplace_merge	par	-9.69 %	++
hpx::inplace_merge	par_unseq	-12.82 %	+++
benchmark_is_heap
hpx::is_heap	par	-18.75 %	+++
benchmark_partial_sort
hpx::partial_sort, size: 1000, step: 1	seq	-8.20 %	++
benchmark_partial_sort_parallel
hpx::partial_sort, size: 1000, step: 1	par	7.43 %	--
hpx::partial_sort, size: 100000	par	2.82 %	-
hpx::partial_sort, size: 100000, step: 1000	par	-11.46 %	+++
benchmark_nth_element_parallel
hpx::nth_element, size: 100000, step: 5000	par	2.86 %	-

…s to 100 Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-24T19:56:05Z

Performance test report

Test name	Executor	Percentage Diff.	Classification
small_vector_benchmark
hpx::small_vector	fxn:void(), 1	-2.24 %	+
hpx::small_vector	fxn:void(), 4	-4.82 %	+
hpx::small_vector	fxn:void(), 8	-3.89 %	+
hpx::small_vector	fxn:void(), 16	-3.05 %	+
future_overhead_report
future overhead - create_thread_hierarchical - latch	no-executor	-1.56 %	+
benchmark_is_heap_until
hpx::heap_until	par	11.67 %	---
benchmark_merge
hpx::merge	par_unseq	2.39 %	-
benchmark_is_heap
hpx::is_heap	par	-18.59 %	+++
benchmark_partial_sort
hpx::partial_sort, size: 1000, step: 1	seq	-4.32 %	+
benchmark_partial_sort_parallel
hpx::partial_sort, size: 1000, step: 1	par	3.59 %	-
hpx::partial_sort, size: 100000	par	4.32 %	-
hpx::partial_sort, size: 100000, step: 1000	par	-4.45 %	+
benchmark_nth_element_parallel
hpx::nth_element, size: 100000, step: 5000	par	2.03 %	-

codacy-production · 2024-08-24T23:52:54Z

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation	Diff coverage
✅ -0.05%	✅ 91.82%

Coverage variation details

	Coverable lines	Covered lines	Coverage
Common ancestor commit (`17bbc63`)	222489	189245	85.06%
Head commit (`c69f9a3`)	196783 (-25706)	167291 (-21954)	85.01% (-0.05%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details

	Coverable lines	Covered lines	Diff coverage
Pull request (#6514)	501	460	91.82%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings Change summary preferences

_{Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more}

…tifacts, and updated baselines to a stable set Signed-off-by: Vedant <[email protected]>

StellarBot · 2024-08-27T20:38:37Z

Performance test report

Test name	Executor	Percentage Diff.	Classification
minmax_element_performance
hpx::minmax	par	-3.11 %	+
small_vector_benchmark
hpx::small_vector	int, 16	2.17 %	-
hpx::small_vector	fxn:void(), 2	3.84 %	-
hpx::small_vector	fxn:void(), 4	3.93 %	-
hpx::small_vector	fxn:void(), 8	3.03 %	-
future_overhead_report
future overhead - create_thread_hierarchical - latch	no-executor	-6.14 %	++
stream_report
Stream benchmark - Add	scheduler_executor_default_scheduler_executor_allocator	-1.73 %	+
Stream benchmark - Triad	scheduler_executor_default_scheduler_executor_allocator	1.83 %	-
benchmark_partial_sort_parallel
hpx::partial_sort, size: 1000, step: 1	par	-2.57 %	+
hpx::partial_sort, size: 100000	par	-2.11 %	+
hpx::partial_sort, size: 100000, step: 1000	par	5.64 %	--

hkaiser added type: enhancement category: CI category: perftests Python performance tests regressions labels Jun 23, 2024

vrnimje added 23 commits July 15, 2024 12:46

Add initial nanobench benchmark config

a105781

Signed-off-by: Vedant <[email protected]>

Pyperf integration for testing puporses

1d9194e

Signed-off-by: Vedant <[email protected]>

Added interface for nanbench backend, and tested interface with minma…

b854128

…x perf test Signed-off-by: Vedant <[email protected]>

Added workflows to test out benchmarks

d21a0c5

Signed-off-by: Vedant <[email protected]>

Modified CI to build and test minmax benchmark only

38e91e3

Signed-off-by: Vedant <[email protected]>

Added separate CI for benchmark without nanobench

7bada72

Signed-off-by: Vedant <[email protected]>

Changing new CI name

8d35c46

Signed-off-by: Vedant <[email protected]>

Correcting CI names

d43015a

Signed-off-by: Vedant <[email protected]>

Changing CI activations to PR only

63d3c59

Signed-off-by: Vedant <[email protected]>

Adding back namespace defs and add_library call for nanobench

2677440

Signed-off-by: Vedant <[email protected]>

Clean up minmax test

296587c

Signed-off-by: Vedant <[email protected]>

Adding command line argument for detailed bench output

4b4ab60

Signed-off-by: Vedant <[email protected]>

Fixing CI format

cb12ce5

Signed-off-by: Vedant <[email protected]>

Adding initial benchmark visualisation script

bf3ad36

Signed-off-by: Vedant <[email protected]>

Added benchmark output specialisations for normal case as well

f7228b9

Signed-off-by: Vedant <[email protected]>

Standardised modules.datastructures.small_vector_benchmark, and made …

4ff833e

…minor changes to plotting script * Further simplified the default template for the benchmark Signed-off-by: Vedant <[email protected]>

Updated cmdline switching to local config, and refactored benchmarks …

acb38b7

…accordingly Signed-off-by: Vedant <[email protected]>

Removing definition of inline variable, and modified CIs to use the c…

95309f9

…orrect option Signed-off-by: Vedant <[email protected]>

Applying cmake-format and clang-format

1ccd4b8

Signed-off-by: Vedant <[email protected]>

Adding explicit double to long double conversions for the default case

da0ce5f

Signed-off-by: Vedant <[email protected]>

Fixed implicit conversion issue while using add_time function

0ede042

Signed-off-by: Vedant <[email protected]>

Standardised foreach_report, stream_report and future_overhead_report…

ceeba99

…, and made minor changes to perftests ci Signed-off-by: Vedant <[email protected]>

Added initial perftests CI config for displaying plots in CDash

7e6606e

Signed-off-by: Vedant <[email protected]>

Fixed subplot indexing issue in perftest plotting script, updated bas…

f6e9b0b

…elines, and fixed clang- and cmake-formatting issues Signed-off-by: Vedant <[email protected]>

Scaling down perftest plot images, so that it is displayed entirely o…

40e38d0

…n CDash, and added them to the build artifacts Signed-off-by: Vedant <[email protected]>

Changed plotting format orientation

54e422c

Signed-off-by: Vedant <[email protected]>

Increasing test output size for CDash, using CTestCustom.ctest file

d7de092

Signed-off-by: Vedant <[email protected]>

Renamed ctestcustom file with cmake extension, and resized image plot…

a9d5df2

…s again Signed-off-by: Vedant <[email protected]>

Increased number of epochs for performance measurements

363eefb

Signed-off-by: Vedant <[email protected]>

Updated baselines with updated number of epochs, and changed uploaded…

195bae6

… image name for CDash Signed-off-by: Vedant <[email protected]>

Separated comparision subplot images, and changed bootstrap statistic…

ff62d46

… to use medians Signed-off-by: Vedant <[email protected]>

Added measurements for each sub-test on CDash, and prefixed performan…

17dfd82

…ce cmdline options with hpx: Signed-off-by: Vedant <[email protected]>

Removed escape characters from the CDash mustache template

f3620d7

Signed-off-by: Vedant <[email protected]>

Added options for test_count in the tests missing it and increased in…

b53373e

…ternal benchmarking iterations to 1000 Signed-off-by: Vedant <[email protected]>

Updated perftest baselines to have 1000 iterations, and cleaned up pe…

7f53069

…rftest_plot script Signed-off-by: Vedant <[email protected]>

Modified bootstrapping parameters, and reduced the perftest iteration…

70461b3

…s to 100 Signed-off-by: Vedant <[email protected]>

Changed plots to histogram with kde, removed generated images from ar…

c69f9a3

…tifacts, and updated baselines to a stable set Signed-off-by: Vedant <[email protected]>

Integration of perftest CI with standardised benchmarks #6514

Are you sure you want to change the base?

Integration of perftest CI with standardised benchmarks #6514

Conversation

vrnimje commented Jun 21, 2024

Proposed Changes

Any background context you want to provide?

jenkins-cscs commented Jun 21, 2024

codacy-production bot commented Jun 21, 2024 • edited Loading

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

vrnimje commented Jul 11, 2024

small_vector_benchmark

stream_report

foreach_report

transform_reduce_scaling

benchmark_is_heap_until

benchmark_inplace_merge

benchmark_is_heap

benchmark_remove

benchmark_remove_if

minmax_element_performance

small_vector_benchmark

stream_report

foreach_report

transform_reduce_scaling

benchmark_is_heap_until

benchmark_inplace_merge

benchmark_is_heap

benchmark_remove

benchmark_remove_if

small_vector_benchmark

stream_report

minmax_element_performance

stream_report

benchmark_remove_if

Pansysk75 commented Jul 11, 2024

StellarBot commented Aug 15, 2024

StellarBot commented Aug 15, 2024

StellarBot commented Aug 15, 2024

StellarBot commented Aug 15, 2024

StellarBot commented Aug 15, 2024

codacy-production bot commented Aug 15, 2024 • edited Loading

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

StellarBot commented Aug 17, 2024

StellarBot commented Aug 17, 2024

StellarBot commented Aug 19, 2024

StellarBot commented Aug 22, 2024

StellarBot commented Aug 22, 2024

StellarBot commented Aug 23, 2024

StellarBot commented Aug 24, 2024

StellarBot commented Aug 24, 2024

codacy-production bot commented Aug 24, 2024 • edited Loading

Coverage summary from Codacy

See diff coverage on Codacy

See your quality gate settings Change summary preferences

StellarBot commented Aug 27, 2024

codacy-production bot commented Jun 21, 2024 •

edited

Loading

codacy-production bot commented Aug 15, 2024 •

edited

Loading

codacy-production bot commented Aug 24, 2024 •

edited

Loading