Skip to content

Setting up a new Deployment of the Upload Service

Jump to bottom
Sam Pierson edited this page Aug 29, 2018 · 4 revisions

Setting up a new Deployment of the Upload Service

Table of Contents

  • Introduction
  • Prerequisites
  • Setup AWS Account
  • Pick and Register Umbrella Domain Name
  • Pick Bucket Prefix
  • Pick Deployment Stage
  • Create SSL Certificate
  • Pick AWS Region
  • Pick VPC CIDR Block
  • Create EC2 Key Pair
  • Create AMI for Batch Validation Workers
  • Create Checksummer Docker Image
  • Create Validator Base Docker Image
  • DCP Ingest AMQP Endpoint
  • Configuration So Far
  • Setup your Environment
  • Setup Terraform
  • Terraform
  • Setup DNS
  • Install and Setup the DCP CLI
  • Testing
  • Subsequent Deploys
  • Teardown

Introduction

This document describes how to setup a new deployment of the Upload Service. There are several different factors that we will address here:

  • Setting up a new domain (i.e. not under .data.humancellatlas.org)
  • Setting up in a new AWS account
  • Setting up a new deployment stage (i.e. other than dev / integration / staging / prod)

Depending on your situation you might need to address one, two or all three of these factors. This document will try to cover the comprehensive case where you are doing all three of the above. Each major heading will be suffixed with letters D, A, S or * in square brackets to indicate this step should be executed for Domain, Account, New Stage or all. Omit steps as necessary if some of these factors do not apply to you.

Prerequisites

  • A Linux / Unix / OS X machine to deploy from.
  • Installed:
    • Python 3.6
    • Docker (e.g. Docker for Mac)
    • terraform (Mac: brew install terraform)
    • jq (Mac: brew install jq)
    • envsubst (Mac: brew install gettext, add /usr/local/opt/gettext/bin to your path)
  • Check out the upload-service repo
  • Install packages: pip install -r requirements-dev.txt

Setup AWS Account [A]

Your AWS account needs to have completed activation and setup billing. You need an IAM user with: * Console and Programmatic access (access keys). * Attached policy AdministratorAccess. * Access keys.

Install the AWS CLI on your local machine and configure using the above access keys, e.g.

pip install awscli
aws configure

Pick and Register Umbrella Domain Name [D]

If you’re deploying under a new DNS domain, e.g. you are forking the DCP for your own purposes, you have some decisions to make.

Decide upon a domain name for your DCP deployment. The DCP contains several services that coordinate by finding each other under a central umbrella domain name. It is also the convention that the pre-production deployments have the stage name inserted between the service name and the domain. For example the umbrella domain for the Human Cell Atlas DCP is data.humancellatlas.org. Production services live directly under this domain:

ingest.data.humancellatlas.org
upload.data.humancellatlas.org
pipelines.data.humancellatlas.org
etc…

The pre-production “staging” deployment stage services live under .staging. at:

ingest.staging.data.humancellatlas.org
upload.staging.data.humancellatlas.org
pipelines.staging.data.humancellatlas.org

Example: in this document, we will assume we are setting up a under a new domain bogodata.org. Production services will live at *.bogodata.org and pre-production services will live at *.<stage>.bogodata.org.

Register your domain name and decide where your DNS records will be hosted. We recommend using Amazon Route53 for this purpose.

Pick Bucket Prefix [D]

Decide on a naming strategy (prefix) for the Upload Service S3 buckets. This should generate globally unique names, e.g the HCA uses its DNS domain in reverse order: org-humancellatlas-upload-. Note the dash on the end of the prefix. The deployment stage name will be appended to this prefix to generate the actual bucket name.

Example: org-bogodata-upload-

Pick Deployment Stage [D,S]

Decide upon the “deployment stage” of your deployment. You may either be performing a production (prod) deployment, or a pre-production deployment, e.g. dev, staging or foo. Only the prod stage has special meaning in the deployment scripts (because of the different DNS implications). Pre-production stages can have any name.

Example: in this document, we will assume we are setting up a new deployment called demo.

Note that when setting up DNS records for this deployment, DNS does not require you to create a new zone (SOA) for a subdomain such as demo.bogodata.org. You can just provide hostnames like upload.demo in your CNAME records under zone bogodata.org.

Create SSL Certificate [D,S]

If you are setting up a new deployment or domain, you will need to source SSL certificate(s) for the deployment-domain combination. I find it easiest to get a free wildcard certificate from the AWS Certificate Manager (ACM). If you don’t use ACM, you will still need to import your certificate into it.

Example: get wildcard SSL certificate for *.demo.bogodata.org

Pick AWS Region [*]

Decide which AWS region you are going to use. Note that some work is already done for you if you use us-east-1. See section “AMI for Batch Validation Workers” below for details.

Actually, looking at the Terraform config (modules/database/database.tf), today only us-east-1 is supported.

Pick VPC CIDR Block [A]

The Upload Service Terraform configuration will create a VPC for you, but you must select the IP address range for the VPC network in CIDR format, e.g. 172.16.0.0/16. You must pick a network that does not overlap with any of the other VPCs in your account.

Create EC2 Key Pair [*]

Setup an EC2 key pair. This key pair will be used when booting Batch workers, and may be used to log in to them to debug them. Either create a key pair or upload a public SSH key to AWS Console -> Services -> EC2 -> Key Pairs. I suggest naming your key pair upload-<stage>, e.g. upload-demo.

Create AMI for Batch Validation Workers [*]

Decide what Amazon Machine Image (AMI) to use for your Batch workers that perform validation. You need an ECS optimized AMI that has sufficient storage space for the size of the files you will be validating. The HCA provides an AMI in us-east-1 region that allocates a 1 TB volume for this purpose: ami-f7a6ca8d (ecso-with-data-vol-v2). You can either:

  • use the ami-f7a6ca8d AMI if you are using region us-east-1
  • copy that AMI to another region if you are using a different region (see Copying an AMI - Amazon Elastic Compute Cloud).
  • build your own AMI using the instructions in amis/README.md

Note the ID of your API.

Create Checksummer Docker Image

::TBD::

Create Validation Docker Image

::TBD::

DCP Ingest AMQP Endpoint

::TBD::

Configuration So Far

So far in our example we have gathered the following information:

  • Domain umbrella: bogodata.org
  • Bucket prefix: org-bogodata-upload-
  • Deployment Stage: demo
  • AWS Region: us-east-1
  • VPC CIDR Block: 172.16.0.0/16
  • Default VPC Security Group ID: sg-xxxxxxxxxxxxxxxxx
  • Key pair for batch workers: upload-demo
  • validation cluster AMI ID: ami-f7a6ca8d

Setup your Environment

Edit config/environment:

vi config/environment
# Changes values:
AWS_DEFAULT_REGION="us-east-1"
DCP_DNS_DOMAIN="bogodata.org"
BUCKET_NAME_PREFIX="org-bogadata-"
# Save it.

echo "export INGEST_API_KEY=long-random-string-i-just-generated" > config/deployment_secrets.dev

export DEPLOYMENT_STAGE=demo
source config/environment
source config/deployment_secrets.demo

Edit config/database.ini and create a stanza for demo (copy the one for dev).

Setup Terraform [A]

Create an S3 bucket where you will store your terraform state:

aws s3 mb s3://org-bogodata-upload-infra

Create a Terraform Configuration for your Deployment

cd terraform/envs
mkdir demo
cp dev/Makefile dev/main.tf demo
cd demo
ln -s ../dev/variables.tf
ln -s ../dev/terraform.tfvars.example

Edit Makefile and set DEPLOYMENT_STAGE and TF_STATE_BUCKET:

DEPLOYMENT_STAGE=demo
TF_STATE_BUCKET=org-bogodata-upload-infra

Edit main.tf and fill in bucket, region and profile where appropriate.

profile is the name of the profile from ~/.aws/config that has access to the terraform state bucket, and the deployment, respectively. This could be default if you only have one account and IAM user.

terraform {
  required_version = "=0.11.7"

  backend "s3" {
    bucket  = "org-bogodata-upload-infra"            // <----
    key     = "terraform/envs/demo/state.tfstate"
    encrypt = true
    region  = "us-east-1"                            // <----
    profile = "bogodata"                             // <----
  }
}

provider "aws" {
  version = ">= 1.31"
  region = "us-east-1"                               // <---- 
  profile = "bogodata"                               // <----
}

Setup terraform.tfvars. Copy the example file:

cp terraform.tfvars.example terraform.tfvars

Then insert all the configuration data we generated in previous steps, together with made-up values for:

  • ingest_api_key
  • db_username only use characters [A-Za-z0-9_]
  • db_password

Initialize Terraform and save the vars file in S3:

make init
make upload-vars

Terraform

Terraforming needs to be broken into a couple of steps the first time, as follows.

First, create the VPC, a bucket to use to deploy Lambdas, and setup the database server and database:

cd terraform/envs/demo

# Preload data used in calculations
terraform apply --target=data.aws_availability_zones.available

# Create VPC
terraform apply --target=module.upload-service.module.upload-vpc.null_resource.vpc

# Create bucket, boot database server, store database secrets
terraform apply --target=module.upload-service.aws_s3_bucket.lambda_area_bucket --target=module.upload-service.module.upload-service-database.aws_secretsmanager_secret_version.database-secrets

# Migrate the database (create tables)
cd ../../..
alembic -x db=${DEPLOYMENT_STAGE} -c=./config/database.ini upgrade head

Next , deploy the API Lambda:

cd chalice
make deploy

Prepare to deploy the Checksum Daemon (this will be done in a moment by Terraform):

cd ../daemons/upload-checksum-daemon

make stage

# Create a dummy version number for now.
aws secretsmanager create-secret --name="dcp/upload/${DEPLOYMENT_STAGE}/upload_service_version" --secret-string='{ "upload_service_version": "'foo'" }'

Complete the terraforming:

cd ../terraform/envs/demo

# Import the API lambda resources you created
tf import module.upload-service.aws_lambda_function.upload_api_lambda upload-api-demo
tf import module.upload-service.aws_iam_role.upload_api_lambda upload-api-demo

make apply

Setup DNS

Chalice created an API Gateway for you, but you need to setup an AWS Custom Domain, and DNS entry.

Go to AWS Console -> Services -> API Gateway -> Custom Domain Names -> Create Custom Domain Name. Fill out the form: Domain Name “upload.demo.bogodata.org” Endpoint Configuration Regional ACM Certificate select the cert you created / imported from the dropdown

Click “Save”. Add a “Base Path Mapping” for “/“; select the API gateway you created from the dropdown, select stage “demo”, click “Save”.

Note the “Target Domain Name”.

Go to Route53 (or your DNS hosting). Create a new record under bogodata.org

upload.demo	CNAME		<target domain name>

Wait a few minutes for DNS to propagate.

If you now point a web browser at http://upload.demo.bogodata.org you should see the API browser.

Install and Setup the DCP CLI

To install (this will likely have been done already when you did pip install -r requirements-dev.txt):

pip install hca

The DCP CLI needs a little tweaking if you are not under the HCA umbrella. Edit ~/.config/hca/config.json (mkdir -p ~/.config/hca if necessary) and tweak the “upload” section to know about your API endpoint URL structure and bucket prefix:

{
	"upload": {
    "preprod_api_url_template": "https://upload.{deployment_stage}.bogodata.org/v1",
    "production_api_url": "https://upload.bogodata.org/v1",
    "bucket_name_template": "org-bogodata-upload-{deployment_stage}",
	}
}

The hca script that is installed may be renamed to something else without affecting its behavior.

Testing

To test your deployment run:

make functional-tests

Subsequent Deploys

# This command is very noisy.  You can ignore the warnings.
scripts/deploy.sh

Teardown

To undo all this configuration, terraform destroy gets rid of most things, but there is a little extra work to do:’

  • Empty your buckets first. Terraform can’t delete non-empty buckets.
  • Delete the DNS record.
  • Delete the API Gateway Custom Domain Name and API Gateway.
  • terraform destroy
  • If you are going to repeat this exercise, really delete your secrets, otherwise Terraform gets confused. Find out the ARN of each of your secrets (deleted or otherwise) then:
aws secretsmanager delete-secret --secret-id="<arn>" --force-delete-without-recovery
  • Delete the *-upload-infra bucket.
Clone this wiki locally