Github Actions is continuous integration/ continous development platform that runs a set of commands on a repository. Github Actions allows the creation of user created modules that automates certain commonly used workflows. On each push, the workflow is run in a docker container (running ubuntu).
- Github Actions allows tests to be run on their servers as such there is no server maintenance required
- There is less security risks because its hosted on the cloud rather than an active server
- Easier local testing allowing for new features to be tested easier
- Flexibility to to tailor the reports to the Einstein Toolkit since we can design our own parsers and tools.
- Larger community giving more opportunity for more plugins than Jenkins.
In orer to make workflow function correctly, a secret needs to be stored on this repo with the name PERSONAL_TOKEN. In order to create the token follow this guide: https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token. After creating the token it can be stored as shown here https://docs.github.com/en/actions/reference/encrypted-secrets.
master:
main.yml- Executes the workflow
scripts:
build-and-test.sh- Compiles and runs the testslogparser.py- Parses the log fileslogpage.py- Generates the HTML pagesstore.py- Stores logs for future usemail.py- Send email each time tests are runregenerate.py- Used to regenerate all HTML pages when new design is createdcreate_csv.py- Used to recreate CSV files when new columns are added
gh-pages:
test_nums.csv- Stores summary stats from logsrecords/- Folder contains compilation logs, logs with summary of tests, and individual test logs and diffs.docs/index.html- HTML page that is displayed on mojamil.github.io/einsteintoolkit/
This file contains the steps for the workflow that need to be executed. It checks if any jobs are running currently, sets up the environment, and pushes all the generated log and html files to the repository.
The workflow is run on each push and can also be run manually
The first part of the workflow is checking if there is already a workflow run in progress, and if that is true, this new run is put into a pending state. If there exists a pending run and the workflow is triggered, the pending workflow is cancelled, and the new workflow has pending status. We use github concurrency to acheive this: https://docs.github.com/en/actions/using-jobs/using-concurrency
The CI runner checks-out both the scripts branch (contains the scripts are to parse and output the data) and the gh-pages branch (contains test log files and HTML output).
It is important to note that for every workflow run, a link to the master branch and gh-pages branch have to be passed. This done to minimize hardcoding of the workflow and help improve transferability of the testing framework.
Then, all the required libraries are installed, after which a clone of the master branch is created and the files from the gh-pages and scripts branches are copied over into a new repository on GitHub's cloud and run:
Then the files with the data that needs to be stored are copied back and pushed to the remote gh-pages branch.
In order check if there was a workflow run that was cancelled the workflow checks if there were in any changes made to the repository and if so it runs the workflow again. This workflow would be run again using this plugin: https://github.com/benc-uk/workflow-dispatch
This python script is used to parse the log files for required data. The parsing is done by using regex expression matching to find the necessary information. Initially, logparser looks through the arguments provided for the master and gh-pages branch.
A brief description of what each function does:
create_summary(file) This function looks for the summary of the tests stored in log files such
as build__2_1.log or build__1_2.log:
get_tests(file) Gets the name of the test that passed and failed as listed in log files such
as build__2_1.log or build__1_2.log:
test_comp(file1,file2) Compares the passed and failed tests from file1 and file2 and returns
which files are newly passing,newly failing, newly added and removed.
get_times(file) This function finds the times taken for each test in the log
file and then stores that in a dictionary and then sorts those tests in descending order by time
exceed_thresh(time_dict,thresh) This function finds tests that exceed a certain time threshhold
longest_tests(time_dict,num_tests) This function uses output from get_times i.e. time_dict to find
num_tests number of the longest test
get_unrunnable(file) This test reads the log file looking for tests that could not be run
and the corresponding reason.
get_data(file) Retrieves singular field of data from a csv and returns it as a list
get_warning_thorns(file) Looks at the compile log and searches for compilation warnings
and outputs the number of warnings per thorn:
get_compile(file) Gets the total number of compilation warnings
get_warning_type(file) Compiles of counts of what types of warnings are produced the most
during compilation
get_warning_thorns(file) This code finds how many compile time warnings are related each thorn
copy_tests(test_dir,version,procs) copies logs and diffs for each test. test_dir is where the test logs and diffs are.The version number and number of procs is used to store the files as shown below:
copy_logs(test_dir, version) This copies the test logs for future use
copy_build(version, test_results, store = None) This copies the old html build files showing test results for future use
copy_compile_log(version) This copies the compilation logs for future use
store_commit_id(version) This stores the current git HEAD hash for future use
get_version(store = None) Gets the version based on the stored files if there are no stored files returns 1
store_version(next_build) This stores the version of the current build in the list of build numbers file.
get_commit_id(version, store = None) This returns the code commit id that this version corresponds to.
Logpage.py generates tables for the html report page and outputs as an html page as shown here: https://einsteintoolkit.github.io/tests/
This file gets the last few commits using githubs REST API for commits and workflow runs as shown in these documentation links: https://docs.github.com/en/rest/reference/repos#commits and https://docs.github.com/en/rest/reference/actions
gen_commits() This function generates a list of commits that have been made since the last run. If the workflow was run manually it will say so.
gen_diffs(readfile) This function generates the html table that shows the comparison of test logs from last version generated by test_comp in logparser
get_first_failure(test_name) This function returns the first version number when this test failed
This table shows the table generated by gen_diffs and get_first_failure:
gen_time(readfile) This function generates a table with the tests that took the longest time
Before you look into the functions that generate plots, it is important to understand bokeh
This file uses bokeh, a python library, to generate plots. The plots are created using python code and bokeh then converts to javascript and html.
Bokeh's plotting works similar to other plotting libraries. First a figure is generated and attributes can be added such as tools to zoom, labels, axis ranges, etc. Bokeh plots using glyphs i.e. given data it will plot it in the format specified for example p.line shown above generates a line graph and p.circle can be used for scatter plots. Bokeh can show its plot locally and save it as a file or generate html and javascript for the plot as shown below:
plot_test_data(readfile) This gets data from the csv and creates lists for each field
gen_unrunnable(readfile) This function generates a html showing which tests could not be run and the reason
create_sidebar() This creates sidebar.html containing all build numbers that gets injected into the HTML page created in summary_to_html
#there is a badge on the einstein toolkit website. Document on how to add this
create_test_results(readfile) This creates test results for the given log file, which gets displayed to the right of the sidebar
summary_to_html(readfile,writefile) This function reads the log file and outputs and html page with the summary in a table
write_to_csv(readfile) This function is used to store data between builds into a csv
This python script is used to regenerate all HTML builds in the case of design changes. This file is similar to logpage. These functions are slightly modified so that the design changes can be run in a for-loop, and hence affect all builds up to date
This python script is used to add columns and automate the creation of a new csv file with previous and newly added data. This file is also similar to logpage. However, since we are only modifying the CSV file, we do not use logpage functions that modify the HTML site.