Skip to content

Benchmarking Platform

Jump to bottom
Patrick Jattke edited this page Aug 6, 2020 · 11 revisions

System Overview

This benchmarking platform is based on Docker and uses AWS EC2 instances to run benchmarks. Also GitHub Actions (GA) are involved to trigger the task of building and publishing Docker images and preparing the benchmarking VMs.

The system is based on two Github Action workflows that are described in the subsequent sections more in detail:

Credentials to AWS, Docker, and also the private deployment key for this repository are passed using the Secrets datastore of this repository.

Folder Structure

The following explains the folder structure of this repository by the example of SEAL. Only files that are expected to be present for every tool are described.

└── SEAL                      <<< main folder, one for each tool
    ├── Dockerfile            <<< Docker image based on image_base/Dockerfile + evaluation programs (source/ directory)
    ├── README.md             <<< additional information (if required)
    ├── image_base            <<< folder for base Docker image 
    │   ├── Dockerfile        <<< Docker image containing SEAL + all required dependencies (is pushed to DockerHub)
    └── source                <<< the evaluation programs adapted to work with SEAL (are copied into container)
        ├── CMakeLists.txt
        └── laplacian_sharpening
            ├── laplacian_sharpening_batched.cpp
            └── laplacian_sharpening_batched.h

Workflow: build_docker_images

Workflow file: docker.yml

The workflow is executed every time any Dockerfile in a image_base/ directory is added or modified. It builds all defined Dockerfiles and pushes the images to the given repository at DockerHub. The workflow must be extended every time a new base image is added by copying the step that uses the mr-smithers-excellent/docker-build-push@v3 action and adapting image and dockerfile appropriately.

Workflow: run_benchmarks

Workflow file: benchmark.yml

The workflow defines several variables (see env: section) that are available to all jobs and their steps. For example, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY that once defined are automatically used by the AWS CLI as credentials.

The workflow consists of three jobs: setup, run-benchmarks, and generate-plots that are described following.

Job: setup

This job generates a timestamp in the format YYYYMMDD_HHMMSS that is used for the name of the folder in the S3 bucket. The timestamp is made available to all subsequent jobs to ensure that all save their results in the same folder. For that, two things are important: First, the timestamp is mapped from the step output to the job output, otherwise it cannot be accessed. Second, the run-benchmarks job specifies the setup job as a dependency, i.e., starts after the setup job finished.

Job: run-benchmarks

First, the GitHub Action creates a new EC2 instance by using the AWS CLI that is pre-installed in the ubuntu-20.04 runner. The step then waits (blocking) until the instance is ready to continue with the next step. The command echo "##[set-output name=instance_id;]$instanceId" makes the instance ID accessible via ${{ steps.container_creation.outputs.instance_id }} in subsequent steps.

Instead of SSH, the GitHub Action uses the AWS Systems Manager Agent (SSM) to communicate with the VM. This has the advantage of not requiring any SSH key but instead only the AWS token that is required anyway. The step then adds the SoK repository SSH deployment key, clones the repository, and starts the respective evaluation docker container.

For ensuring that the VM is shut down in case of an error, we execute the command echo "sudo halt" | at now + ${{ env.instance-termination-timeout }} minutes that automatically shuts down the VM after the amount of minutes specified in the instance-termination-timeout workflow variable. Due to the defined termination behavior of the VM, the VM is terminated in case of a shutdown instead of stopped only.

Job: generate-plots

This job waits until all benchmarks were executed and then copies all plotting files (*.py) from the repository folder (scripts/plotting/) to the plot/ folder in the respective S3 benchmark folder (e.g., 20200804_160933/plot) of the current run.

As AWS SSM just sends commands to the VM, i.e., the workflow does not know if the benchmark finished, there is a (blocking) waiting mechanism implemented. This mechanism checks the VM's state each minute using the AWS CLI and aborts if the value defined by the workflow variable instance-termination-timeout (e.g., 30 minutes) is reached. In that case, the workflow continues with the job's execution.

Then, the workflow installs Python and the dependencies by taking the requirements.txt. Finally, the plot_all.py is executed that generates and uploads the plots to S3.

AWS

This section quickly summarizes the required AWS configuration to run the GAs.

IAM

The GitHub Action in the run_benchmarks workflow uses a AWS user to spin up the EC2 VMs. Also, the evaluation docker image (the one that includes the evaluation programs) uses the same user to upload the results to a S3 bucket. As such, the AWS user sok-repository-github-action requires the following permissions:

- AmazonEC2FullAccess (AWS managed policy)   – for creating and running EC2 VMs
- AmazonSSMFullAccess (AWS managed policy)   - for accessing the VM via SSM
- AddRoleToInstance (managed policy)         - for assigning the AmazonSSMFullAccess role to the EC2 VM
- AmazonS3FullAccess (AWS managed policy)    - for listing and write access to the sok-repository-eval-benchmarks S3 bucket

For reproducibility, the policy summaries of the managed policies are given following.

AddRoleToInstance

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ec2:AssociateIamInstanceProfile",
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::400029883903:role/ec2-amazonSSMFullAccess"
        }
    ]
}

Amazon Machine Image (AMI)

A new AMI (ID: ami-0da952e8afc41613f) was created based on the official Ubuntu 20.04 LTS - Focal by Canonical.

Modifications made:

  • Docker installed

Updates to be made for the next AMI release

  • Add default ubuntu user to docker group: 
    sudo usermod -aG docker $USER
newgrp docker
    This is not an issue at the moment as AWS SSM uses the root account anyway.

S3

The S3 bucket's URL is s3://sok-repository-eval-benchmarks. The bucket contains a directory for each workflow run that is named according to the start date/time of the run-benchmarks workflow. This timestamp is generated by the setup job in the workflow.

The following shows the content of the bucket as an example:

─── 20200805_181943
│   └── ...
├── 20200805_201905
│   └── ...
├── 20200805_203445
│   ├── ...
├── 20200806_001631                // benchmark started at August 6, 2020 at 00:16:31
│   ├── ALCHEMY                    // tool folder
│   │   └── alchemy_cardio.csv     // result for benchmark program 'cardio'
│   ├── Cingulata
│   │   └── cingulata_cardio.csv
│   ├── Lobster
│   │   ├── lobster_cardio.csv
│   ├── SEAL
│   │   └── seal_cardio.csv
│   ├── ..
│   └── plot                       // plot scripts + plots generated by workflow
│       ├── plot_all.py
│       ├── plot_cardio.pdf
│       ├── plot_cardio.png
│       ├── plot_cardio.py
│       ├── requirements.txt
│       └── s3_utils.py

Run Benchmarks

The benchmarks can be executed by triggering the repository_dispatch event. This event cannot be executed via the GitHub website. A POST request must be invoked, for example, via cURL. For that, it is require to authenticate to the GitHub API using a personal access token, which can be created in the Developer Settings of your profile. Access permissions for the repo scope is required for the token.

via cURL

Replace your-username with your GitHub username and your-personal-access-token with your previously generated access token.

curl -X POST https://api.github.com/repos/MarbleHE/SoK/dispatches -H 'Accept: application/vnd.github.everest-preview+json' -u your-username:your-personal-access-token --data '{"event_type": "run_benchmarks"}'

via VSCode

  1. Install the GitHub Actions extension
  2. When prompted for a PAT, enter your personal access token.
  3. Right-click on the workflow and choose trigger workflow (see the workflow's description page for details).

Status Monitoring

The progress of the workflow execution can be observed on the workflow's status page. Note that the workflow finishes after sending the respective commands to the created EC2 instances, i.e., it does not not wait until the command execution finished.

At the end of the workflow execution, a command ID and a AWS CLI command is printed. This can be used to check the SSM task progress and its output is given as an example in the following:

> aws ssm list-command-invocations --command-id ad7a56c1-20f1-4413-b4eb-407171711a04 --details
{
    "CommandInvocations": [
        {
            "CommandId": "ad7a56c1-20f1-4413-b4eb-407171711a04",
            "InstanceId": "i-02c20258e4880cfe6",
            "InstanceName": "",
            "Comment": "Executed by Github Actions",
            "DocumentName": "AWS-RunShellScript",
            "DocumentVersion": "",
            "RequestedDateTime": "2020-07-20T15:55:04.416000+02:00",
            "Status": "InProgress",                     <<< shows the current status of command execution
            "StatusDetails": "InProgress",
            "StandardOutputUrl": "",
            "StandardErrorUrl": "",
            "CommandPlugins": [
                {
                    "Name": "aws:runShellScript",
                    "Status": "InProgress",
                    "StatusDetails": "InProgress",
                    "ResponseCode": -1,
                    "Output": "",                        <<< shows the output of the executed command
                    "StandardOutputUrl": "",
                    "StandardErrorUrl": "",
                    "OutputS3Region": "",
                    "OutputS3BucketName": "",
                    "OutputS3KeyPrefix": ""
                }
            ],
            "ServiceRole": "",
            "NotificationConfig": {
                "NotificationArn": "",
                "NotificationEvents": [],
                "NotificationType": ""
            },
            "CloudWatchOutputConfig": {
                "CloudWatchLogGroupName": "",
                "CloudWatchOutputEnabled": false
            }
        }
    ]
}

Retrieving Results

The results must manually be retrieved from the S3 bucket sok-repository-eval-benchmarks.

The subfolder plot/ in a workflow's run folder contains plots generated during the workflow as PDF and PNG.

The interactive visualization website is currently work in progress, see Visualization

Clone this wiki locally