Redshift Cluster via Terraform

Table of Contents

• Redshift Cluster via Terraform
  • Overview
    • Terraform
      • Quick Start
      • Details
    • Boto3
  • Architecture
  • Structure

Overview

The purpose of this repo is to provide an easy method to create, update and destroy redshift clusters via Terraform.

The redshift cluster will also be able to access AWS S3 buckets.

Terraform

Quick Start

Show/Hide Details

Ensure you have terraform cli installed and setup for local development. More information here

Rename terraform.tfvars.example as terraform.tfvars. Update as appropriate as this will contain the value for all relevant variables.

Run terraform init to initialise the terraform directory.

Run terraform plan to view the execution plan.

Run terraform apply to create the redshift cluster. Make sure to enter yes to confirm the execution.

Finally to delete the cluster run terraform destroy. Make sure to enter yes to confim the deletion and to destroy any other related objects.

Details

When you run terraform apply the following resources are being created:

1. aws_iam_role: This is an IAM role that will be provided to Redshift to allow it to access other AWS resources
2. aws_iam_role_policy_attachment: this will attach an s3 read access policy to the newly created IAM role. This will enable the redshift cluster to access S3 buckets.
3. aws_redshift_cluster: The Amazon Redshift Cluster
4. aws_vpc: An Amazon Virtual Private Cloud (VPC)
5. aws_default_security_group: Opens a TCP connection within the VPC at the specified port. This will enable you to query the Redshift database at your provided IP address

Boto3

Show/Hide Details

For reference I have provided python scripts to create a redshift cluster via boto3.

In your environment of choice install the following packages using python 3.6.10:

• boto3==1.18.9
• psycopg2==2.9.1

Rename dwh.cfg.example as dwh.cfg. Update as appropriate as this will contain the value for all relevant variables.

Within the scripts/ folder run: python create_redshift_cluster False to create a redshift cluster. The first argument passed referes to an option to delete the redshift cluster.

Finally to delete the cluster run python create_redshift_cluster True to delete the redshift cluster.

Architecture

Terraform is an IaC tool that allows engineers / developers to build, change and destroy infrastructure conveniently and efficiently.

Amazon Redshift is a column oriented data warehouse with a Postgres compatible querying layer. With Redshift it is possible to scale from 1 to N clusters depending on your storage and query performance needs.

Structure

Show/Hide Details

• main.tf: AWS resources to be created by Terraform
• outputs.tf: output values once Terraform has created all necessary resources
• scripts/: python scripts to crreate redshift cluster via boto3
  • create_redshift_cluster.py: create redshift cluster via boto3
  • dwh.cfg.example: example config to create redshift cluster via boto3
• terraform.tfstate: state file to keep track of resources created by Terraform configuration
• terraform.tfvars.example: value of all input variables
• variable.tf: holds description and default for all input varaibles