Skip to content

Benchmarking Platform

Jump to bottom
Patrick Jattke edited this page Jul 28, 2020 · 11 revisions

System Overview

This benchmarking platform is based on Docker and uses AWS EC2 instances to run benchmarks. Also GitHub Actions (GA) are involved to trigger the task of building and publishing Docker images and preparing the benchmarking VMs.

The system is based on two Github Action workflows that are described in the subsequent sections more in detail:

Credentials to AWS, Docker, and also the private deployment key for this repository are passed using the Secrets datastore of this repository.

Folder Structure

The following explains the folder structure of this repository by the example of SEAL. Only files that are expected to be present for every tool are described.

├── README.md
└── SEAL                      <<< main folder, one for each tool
    ├── Dockerfile            <<< Docker image based on image_base/Dockerfile plus evaluation programs (source/ directory)
    ├── README.md             <<< additional information (if required)
    ├── image_base            <<< folder for base Docker image 
    │   ├── Dockerfile        <<< Docker image containing SEAL + all required dependencies (is pushed to DockerHub)
    └── source                <<< the evaluation programs adapted to work with SEAL
        ├── CMakeLists.txt
        └── laplacian_sharpening
            ├── laplacian_sharpening_batched.cpp
            └── laplacian_sharpening_batched.h

Workflow: build_docker_images

Workflow file: docker.yml

The workflow is executed every time any Dockerfile in a image_base/ directory is added or modified. It builds all defined Dockerfiles and pushes the images to the given repository at DockerHub. The workflow must be extended every time a new base image is added by copying the step that uses the mr-smithers-excellent/docker-build-push@v3 action and adapting image and dockerfile appropriately.

Workflow: run_benchmarks

Workflow file: benchmark.yml

// TODO: Finish explanation

First, the GitHub Action creates a new EC2 instance by using the AWS CLI that is pre-installed in the ubuntu-20.04 runner. The step then waits (blocking) until the instance is ready to continue with the next step. The command echo "##[set-output name=instance_id;]$instanceId" makes the instance ID accessible via ${{ steps.container_creation.outputs.instance_id }} in subsequent steps.

Instead of SSH, the GitHub Action uses the AWS Systems Manager Agent (SSM) to communicate with the VM. This has the advantage of not requiring any SSH key but instead only the AWS token that is required anyway. The step then

The job run-benchmark-seal must be copied and adapted for each newly added tool. It is important to ensure that the --instance-type is the same for all benchmarked tools.

AWS

This section quickly summarizes the required AWS configuration to run the GAs.

IAM

The GitHub Action in the run_benchmarks workflow uses a AWS user to spin up the EC2 VMs. Also, the evaluation docker image (the one that includes the evaluation programs) uses the same user to upload the results to a S3 bucket. As such, the AWS user sok-repository-github-action requires the following permissions:

- AmazonEC2FullAccess (AWS managed policy)   – for creating and running EC2 VMs
- AmazonSSMFullAccess (AWS managed policy)   - for accessing the VM via SSM
- AddRoleToInstance (managed policy)         - for assigning the AmazonSSMFullAccess role to the EC2 VM
- S3-PutObject (managed policy)              - for write access to the sok-repository-eval-benchmarks S3 bucket

For reproducibility, the policy summaries of the managed policies are given following.

AddRoleToInstance

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "ec2:AssociateIamInstanceProfile",
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::400029883903:role/ec2-amazonSSMFullAccess"
        }
    ]
}

S3-PutObject

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::sok-repository-eval-benchmarks/*"
        }
    ]
}

Amazon Machine Image (AMI)

A new AMI (ID: ami-0da952e8afc41613f) was created based on the official Ubuntu 20.04 LTS - Focal by Canonical.

Modifications made:

  • Docker installed

S3

// TODO: Finish explanation

Run Benchmarks

The benchmarks can be executed by triggering the repository_dispatch event. This event cannot be executed via the GitHub website. A POST request must be invoked, for example, via cURL. For that, it is require to authenticate to the GitHub API using a personal access token, which can be created in the Developer Settings of your profile. Access permissions for the repo scope is required for the token.

via cURL

Replace your-username with your GitHub username and your-personal-access-token with your previously generated access token.

curl -X POST https://api.github.com/repos/MarbleHE/SoK/dispatches -H 'Accept: application/vnd.github.everest-preview+json' -u your-username:your-personal-access-token --data '{"event_type": "run_benchmarks"}'

via VSCode

  1. Install the GitHub Actions extension
  2. When prompted for a PAT, enter your personal access token.
  3. Right-click on the workflow and choose trigger workflow (see the workflow's description page for details).

Status Monitoring

The progress of the workflow execution can be observed on the workflow's status page. Note that the workflow finishes after sending the respective commands to the created EC2 instances, i.e., it does not not wait until the command execution finished.

At the end of the workflow execution, a command ID and a AWS CLI command is printed. This can be used to check the SSM task progress and its output is given as an example in the following:

> aws ssm list-command-invocations --command-id ad7a56c1-20f1-4413-b4eb-407171711a04 --details
{
    "CommandInvocations": [
        {
            "CommandId": "ad7a56c1-20f1-4413-b4eb-407171711a04",
            "InstanceId": "i-02c20258e4880cfe6",
            "InstanceName": "",
            "Comment": "Executed by Github Actions",
            "DocumentName": "AWS-RunShellScript",
            "DocumentVersion": "",
            "RequestedDateTime": "2020-07-20T15:55:04.416000+02:00",
            "Status": "InProgress",                     <<< shows the current status of command execution
            "StatusDetails": "InProgress",
            "StandardOutputUrl": "",
            "StandardErrorUrl": "",
            "CommandPlugins": [
                {
                    "Name": "aws:runShellScript",
                    "Status": "InProgress",
                    "StatusDetails": "InProgress",
                    "ResponseCode": -1,
                    "Output": "",                        <<< shows the output of the executed command
                    "StandardOutputUrl": "",
                    "StandardErrorUrl": "",
                    "OutputS3Region": "",
                    "OutputS3BucketName": "",
                    "OutputS3KeyPrefix": ""
                }
            ],
            "ServiceRole": "",
            "NotificationConfig": {
                "NotificationArn": "",
                "NotificationEvents": [],
                "NotificationType": ""
            },
            "CloudWatchOutputConfig": {
                "CloudWatchLogGroupName": "",
                "CloudWatchOutputEnabled": false
            }
        }
    ]
}

Retrieving Results

The results must manually be retrieved from the S3 bucket sok-repository-eval-benchmarks.

The interactive visualization website is currently work in progress.

Clone this wiki locally