Continuous benchmarking using Github Actions

Authors

• Johan Hammarstedt ([email protected]), Github: jhammarstedt
• Carl Leijonberg ([email protected]), Github : carllei

Task

This project was aimed to teach others how to set up a Github Action to create continuous benchmarking with pytest. We also added a simple visualization with Github Pages that we walk through briefly in the tutorial.

This will help developers easily compare benchmark results and alert on worse performance when making new commits. The statistics from the latest run are found in output.json and the historical comparison table is visualized in the generated page available here.

Katacoda tutorial

We have created a katacoda tutorial that runs a bash terminal and a VS code environment in the browser. It walks through every step how to build and set up this repo yourself.

It's available here!💡

You will learn how to:

• Create simple Github Action that will let you test and compare python scripts on pushes to your GitHub repository
  • With a few modifications, you can also implement them for other tasks to enable CI/CD in your other projects!
  • You can also use this action with similar performance tools for other programming languages.
• Create your first Github Page that will display the results from your testing

Description and usage (If you're not planning on doing the tutorial)

To try this project out simply fork this repository and create a commit.

# Clone the repository
$ git clone https://github.com/jhammarstedt/Benchmarking-DevOps.git
$ cd Benchmarking-DevOps
# Clear the table
$ chmod +x ./scripts/clear_table.sh
$ ./scripts/clear_table.sh

To make the workflow run commit these changes to your repository.

Compare our test functions or write your own in the benchmarking.py script script. We have 2 test functions, a slower turtle and faster cheetah which are just used to test the benchmarking and display some results in the workflow and on your github page.

• The workflow file is found in .github/workflows/python.yml
• The configurations for the webpage is found in docs
• All python scripts are found in src

Enable Github Pages

1. Go to settings for your cloned repository
2. Find pages (almost all the way down to the left)
3. The source should be master and also specify /docs as there is an HTML file running it.
4. Press the link generated (might take up to 30 sec after enabling it)

Clean the table in Gh Pages

If you want to reset the table again simply run $ ./scripts/clear_table.sh

Easter egg hint for tutorial

Click me for hint

Did you collect the 🥚 from scripts?

Fun fact regarding easter egg (open after finding it)

The author of the action did not support memes by repo owners, which would be a problem for our purpose. So I raised that issue and got a new feature merged for this tutorial 🤙

Future work

• Add support for more languages