Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integration of perftest CI with standardised benchmarks #6514

Draft
wants to merge 75 commits into
base: master
Choose a base branch
from
Draft

Integration of perftest CI with standardised benchmarks #6514

wants to merge 75 commits into from
+59,681 −1,135

Commits on Jul 15, 2024

  1. Add initial nanobench benchmark config 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    a105781 View commit details
    Browse the repository at this point in the history

  2. Pyperf integration for testing puporses 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    1d9194e View commit details
    Browse the repository at this point in the history

  3. Added interface for nanbench backend, and tested interface with minma… 

    …x perf test

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    b854128 View commit details
    Browse the repository at this point in the history

  4. Added workflows to test out benchmarks 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    d21a0c5 View commit details
    Browse the repository at this point in the history

  5. Modified CI to build and test minmax benchmark only 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    38e91e3 View commit details
    Browse the repository at this point in the history

  6. Added separate CI for benchmark without nanobench 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    7bada72 View commit details
    Browse the repository at this point in the history

  7. Changing new CI name 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    8d35c46 View commit details
    Browse the repository at this point in the history

  8. Correcting CI names 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    d43015a View commit details
    Browse the repository at this point in the history

  9. Changing CI activations to PR only 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    63d3c59 View commit details
    Browse the repository at this point in the history

  10. Adding back namespace defs and add_library call for nanobench 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    2677440 View commit details
    Browse the repository at this point in the history

  11. Clean up minmax test 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    296587c View commit details
    Browse the repository at this point in the history

  12. Adding command line argument for detailed bench output 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    4b4ab60 View commit details
    Browse the repository at this point in the history

  13. Fixing CI format 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    cb12ce5 View commit details
    Browse the repository at this point in the history

  14. Adding initial benchmark visualisation script 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    bf3ad36 View commit details
    Browse the repository at this point in the history

  15. Added benchmark output specialisations for normal case as well 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    f7228b9 View commit details
    Browse the repository at this point in the history

  16. Standardised modules.datastructures.small_vector_benchmark, and made … 

    …minor changes to plotting script

* Further simplified the default template for the benchmark

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    4ff833e View commit details
    Browse the repository at this point in the history

  17. Updated cmdline switching to local config, and refactored benchmarks … 

    …accordingly

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    acb38b7 View commit details
    Browse the repository at this point in the history

  18. Removing definition of inline variable, and modified CIs to use the c… 

    …orrect option

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    95309f9 View commit details
    Browse the repository at this point in the history

  19. Applying cmake-format and clang-format 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    1ccd4b8 View commit details
    Browse the repository at this point in the history

  20. Adding explicit double to long double conversions for the default case 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    da0ce5f View commit details
    Browse the repository at this point in the history

  21. Fixed implicit conversion issue while using add_time function 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    0ede042 View commit details
    Browse the repository at this point in the history

  22. Standardised foreach_report, stream_report and future_overhead_report… 

    …, and made minor changes to perftests ci

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    ceeba99 View commit details
    Browse the repository at this point in the history

  23. Added initial perftests CI config for displaying plots in CDash 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    7e6606e View commit details
    Browse the repository at this point in the history

  24. Configured ctest.cmake into CI, instead of directly using ctest -D 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    bab5e13 View commit details
    Browse the repository at this point in the history

  25. Moved plot generation to ctest.cmake 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    b9a0c72 View commit details
    Browse the repository at this point in the history

  26. Changed some internal paths for json file generation 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    bce01c6 View commit details
    Browse the repository at this point in the history

  27. Fixed issue, where python script was not executing from ctest.cmake 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    f6402f8 View commit details
    Browse the repository at this point in the history

  28. Adding custom target for cdash results and plot generation, for each … 

    …perftest

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    7053cc6 View commit details
    Browse the repository at this point in the history

  29. Separate cmake function defined for perftest reports, for taking addi… 

    …tonal params

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    7e51dc8 View commit details
    Browse the repository at this point in the history

  30. Added more perf-report tests, and made improvements to the perftest h… 

    …andling

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    d31811a View commit details
    Browse the repository at this point in the history

  31. Removing debgugging modifications 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    52348b4 View commit details
    Browse the repository at this point in the history

  32. Removed the cdash image printing for detailed config of perftest report 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    a280fe5 View commit details
    Browse the repository at this point in the history

  33. Defined the cmake config command for cdash properly 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    6783468 View commit details
    Browse the repository at this point in the history

  34. Fixing syntax for perftest_init in standardised benchmarks 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    5c813af View commit details
    Browse the repository at this point in the history

  35. Removing unnecessary perftest_cfg call from future_overhead_report test 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    5ee6eb0 View commit details
    Browse the repository at this point in the history

  36. Update the plotting script 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    950c28d View commit details
    Browse the repository at this point in the history

  37. Added comparision with pre-defined baselines, and standardised transf… 

    …orm_reduce_scaling and benchmark_is_heap_until

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    192e943 View commit details
    Browse the repository at this point in the history

  38. Standardised benchmark_inplace_merge and benchmark_merge 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    a27a1f0 View commit details
    Browse the repository at this point in the history

  39. Standardised benchmark_is_heap and benchmark_remove 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    eba0181 View commit details
    Browse the repository at this point in the history

  40. Standardised benchmark_remove_if and fixed formatting issues with ben… 

    …chmark results

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    6b7b969 View commit details
    Browse the repository at this point in the history

  41. Added KS-test for benchmark comparision, and updated baselines 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    8c7c1a6 View commit details
    Browse the repository at this point in the history

  42. Made modifications for perftests comment mechanism 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    6772090 View commit details
    Browse the repository at this point in the history

  43. Removed the reports folder from the artifacts, and clang-formatted files 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    6e4c647 View commit details
    Browse the repository at this point in the history

  44. Moved comment mechanism to entry.sh 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    7dd0bf7 View commit details
    Browse the repository at this point in the history

  45. Sourcing the environment file, and removed boost fetch from perftests… 

    …' ctest

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    2cf0409 View commit details
    Browse the repository at this point in the history

  46. Added asio fetch option for CMake Configure step in perftests' ctest.… 

    …cmake

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    f48016f View commit details
    Browse the repository at this point in the history

  47. Manually installing scipy module for plotting script 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    7100ba0 View commit details
    Browse the repository at this point in the history

  48. Using user installation for scipy, due to permission issues 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    3eac684 View commit details
    Browse the repository at this point in the history

  49. Removed scipy installation, as it has been added to virtual env 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    db6137e View commit details
    Browse the repository at this point in the history

  50. Importing scipy.stats in plotting script 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    b935650 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. Removed pushd commands in comment_github.sh 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    6072f1b View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2024

  1. Re-formatted comment for performance discrepancy, and reverted back t… 

    …o HPX CDash creds

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    1ab4def View commit details
    Browse the repository at this point in the history

  2. Removed addition of newlines into perf-report, and referenced correct… 

    … variables

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    d305da4 View commit details
    Browse the repository at this point in the history

  3. Added commands to write CTest Build ID to the file, in ctest.cmake 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    f197521 View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. Calling status_computation function manually in lsu-perftest, and men… 

    …tioned unused variables in benchmark_is_heap_until

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    d3b59d0 View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. Added option for printing CDash image path, and modified lsu-perftest… 

    … for uploading results JSONs

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    b3e4382 View commit details
    Browse the repository at this point in the history

  2. Updated perftest-baselines, moved comment mechanism to status functio… 

    …n and fixed formatting

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    653a28a View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2024

  1. Removed explicit call to artifacts-storage function, and modified per… 

    …ftest-comment format

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    d561688 View commit details
    Browse the repository at this point in the history

  2. Removing ctest_status check for commenting perftest-report 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    1589588 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. Changed plotting format, and made definitions of dependencies and exe… 

    …cutables for cdash targets implicit

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    9f3853d View commit details
    Browse the repository at this point in the history

  2. Standardised the following tests, and made formatting changes to plot… 

    …ting script

* benchmark_partial_sort
* benchmark_partial_sort_parallel
* benchmark_nth_element
* benchmark_nth_element_parallel

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    91a6d3f View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. Fixed subplot indexing issue in perftest plotting script, updated bas… 

    …elines, and fixed clang- and cmake-formatting issues

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    f6e9b0b View commit details
    Browse the repository at this point in the history

  2. Scaling down perftest plot images, so that it is displayed entirely o… 

    …n CDash, and added them to the build artifacts

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    40e38d0 View commit details
    Browse the repository at this point in the history

  3. Changed plotting format orientation 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    54e422c View commit details
    Browse the repository at this point in the history

  4. Increasing test output size for CDash, using CTestCustom.ctest file 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    d7de092 View commit details
    Browse the repository at this point in the history

  5. Renamed ctestcustom file with cmake extension, and resized image plot… 

    …s again

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    a9d5df2 View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2024

  1. Increased number of epochs for performance measurements 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    363eefb View commit details
    Browse the repository at this point in the history

  2. Updated baselines with updated number of epochs, and changed uploaded… 

    … image name for CDash

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 17, 2024
    Configuration menu
    Copy the full SHA
    195bae6 View commit details
    Browse the repository at this point in the history

Commits on Aug 19, 2024

  1. Separated comparision subplot images, and changed bootstrap statistic… 

    … to use medians

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 19, 2024
    Configuration menu
    Copy the full SHA
    ff62d46 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. Added measurements for each sub-test on CDash, and prefixed performan… 

    …ce cmdline options with hpx:

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    17dfd82 View commit details
    Browse the repository at this point in the history

  2. Removed escape characters from the CDash mustache template 

    Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    f3620d7 View commit details
    Browse the repository at this point in the history

Commits on Aug 23, 2024

  1. Added options for test_count in the tests missing it and increased in… 

    …ternal benchmarking iterations to 1000

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 23, 2024
    Configuration menu
    Copy the full SHA
    b53373e View commit details
    Browse the repository at this point in the history

Commits on Aug 24, 2024

  1. Updated perftest baselines to have 1000 iterations, and cleaned up pe… 

    …rftest_plot script

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 24, 2024
    Configuration menu
    Copy the full SHA
    7f53069 View commit details
    Browse the repository at this point in the history

  2. Modified bootstrapping parameters, and reduced the perftest iteration… 

    …s to 100

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 24, 2024
    Configuration menu
    Copy the full SHA
    70461b3 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. Changed plots to histogram with kde, removed generated images from ar… 

    …tifacts, and updated baselines to a stable set

Signed-off-by: Vedant <[email protected]>
    @vrnimje
    vrnimje committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    c69f9a3 View commit details
    Browse the repository at this point in the history