terraform-polkadot-aws-asg

This module sets up a load balanced set of endpoints for the Polkadot blockchain and it's associated parachains on AWS. Components include:

• AMI produced from a packer build
• Autoscaling group
• Network load balancer
• Scaling policies

The module is intended to be flexible in its configuration parameters allowing users specify networks and securtiy groups while also providing sane defaults for one click deployments. Users then have the option of attaching their own DNS record or with additional configuration, joining to a consul cluster and monitoring with prometheus.

Requirements

• Terraform version 0.14+ tested - Install
• Packer version 1.7+ - Install
• Ansible 2.9 - pip install ansible
• SSH Keys - ssh-keygen -b 4096 (Only public required)

Usage

Steps for running terraform:

1. Install the above requirements

2. Get AWS API keys into environment variables

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

3. Create SSH keys and make note of the path (public_key_path variable) or copy the key material.

ssh-keygen -b 4096 -f $HOME/.ssh/<your key name>
cat $HOME/.ssh/<your key name>.pub # this is the `public_key` variable 

4. Use this module in your own terraform or modify one of the examples directory. Possible configurations are:

Defaults

Minimal defaults example.

module "defaults" {
  source     = "github.com/geometry-labs/terraform-polkadot-aws-asg"
  name       = "some-name"
  public_key = file(var.public_key_path)
}

Deploys in default vpc and creates security group. For public deployments

External Network with Parachains

To run additional parachains, complete the below map for network_settings to map ports to the associated chain. Ports will then be exposed over the load balancer.

locals {
  network_settings = {
    polkadot = {
      name                = "polkadot"
      shortname           = "polkadot"
      api_health          = "5000"
      polkadot_prometheus = "9610"
      json_rpc            = "9933"
      json_envoy          = "21000"
      ws_rpc              = "9944"
      ws_envoy            = "21001"
    }
    kusama = {
      name                = "kusama"
      shortname           = "ksmcc3"
      api_health          = "5001"
      polkadot_prometheus = "9611"
      json_rpc            = "9934"
      json_envoy          = "21002"
      ws_rpc              = "9945"
      ws_envoy            = "21003"
    }
  }
}

module "network" {
  source           = "github.com/geometry-labs/terraform-polkadot-aws-network.git?ref=main"
  api_enabled      = true
  num_azs          = 2
  network_settings = local.network_settings
}

variable "public_key" {}

resource "random_pet" "this" {}

module "defaults" {
  source = "../.."

  name = "external-${random_pet.this.id}"

  create_security_group = false

  public_key      = var.public_key
  security_groups = [module.network.api_security_group_id]
  subnet_ids      = module.network.public_subnets
  vpc_id          = module.network.vpc_id

  min_size         = 1
  max_size         = 1
  desired_capacity = 1

  network_settings = local.network_settings

  depends_on = [module.network]
}

Examples

• defaults

• spot

• external-network

• standalone

Dependencies

• terraform-packer-build
• terraform-polkadot-user-data

Known Issues

• When creating a network within the same module as this one, you need to explicitly put a depends_on on the network to make the graph resolve properly or apply it twice.

Requirements

Name	Version
terraform	>= 0.12

Providers

Name	Version
aws	3.37.0
null	3.1.0
random	3.1.0

Modules

Name	Source	Version
asg	terraform-aws-modules/autoscaling/aws	~> 4.11.0
packer	github.com/geometry-labs/terraform-packer-build.git	v0.1.0
user_data	github.com/geometry-labs/terraform-polkadot-user-data.git	n/a

Resources

Name	Type
aws_autoscaling_policy.this	resource
aws_eip.this	resource
aws_iam_instance_profile.this	resource
aws_iam_policy.describe_policy	resource
aws_iam_policy.sot_kms_decrypt	resource
aws_iam_policy.sot_read	resource
aws_iam_role.this	resource
aws_iam_role_policy_attachment.describe_policy	resource
aws_iam_role_policy_attachment.sot_kms_decrypt	resource
aws_iam_role_policy_attachment.sot_read	resource
aws_key_pair.this	resource
aws_lb.this	resource
aws_lb_listener.ext-health	resource
aws_lb_listener.rpc	resource
aws_lb_listener.wss	resource
aws_lb_target_group.ext-health	resource
aws_lb_target_group.rpc	resource
aws_lb_target_group.wss	resource
aws_security_group.this	resource
aws_security_group_rule.consul_ingress	resource
aws_security_group_rule.egress	resource
aws_security_group_rule.ingress	resource
aws_security_group_rule.public_ingress	resource
null_resource.requirements	resource
null_resource.wait	resource
random_pet.this	resource
aws_ami.packer	data source
aws_iam_policy_document.assume_policy_document	data source
aws_iam_policy_document.describe_policy	data source
aws_iam_policy_document.sot_bucket_read	data source
aws_iam_policy_document.sot_kms_key_read	data source
aws_region.this	data source
aws_subnet_ids.this	data source
aws_vpc.default	data source

Inputs

Name	Description	Type	Default	Required
additional_build_security_group_ids	Additional security groups to use to build image.	list(string)	[ "" ]	no
ami_id	AMI ID to use in autoscaling group. Blank to build from packer.	string	""	no
aws_region	n/a	string	""	no
base_path	Base path for Substrate	string	""	no
boot_drive_nvme	Boolean to set if instance boot drive is also nvme	bool	false	no
build_security_group_id	The security group to use to build image.	string	""	no
build_subnet_id	The subnet to build the image in. Must be public - Omit if running cluster deployed in in public subnets.	string	""	no
build_vpc_id	VPC to build the image in. Must have public subnet - Omit if running cluster deployed in in public subnets.	string	""	no
cluster_name	The name of the k8s cluster	string	""	no
consul_acl_datacenter	Authoritative Consul ACL datacenter	string	""	no
consul_acl_enable	Bool to enable Consul ACLs	bool	false	no
consul_acl_token	Consul ACL token	string	""	no
consul_auto_encrypt_enabled	Bool to enable Consul auto-encrypt	bool	false	no
consul_connect_enabled	Bool to enable Consul Connect	bool	false	no
consul_datacenter	n/a	string	""	no
consul_enabled	Bool to use when Consul is enabled	bool	false	no
consul_gossip_key	Consul gossip encryption key	string	""	no
consul_security_group	ID of security group to containing Consul	string	null	no
consul_tls_ca_crt	n/a	string	""	no
consul_tls_ca_filename	Filename of Consul CA	string	""	no
consul_tls_source_dir	Path to TLS files	string	""	no
consul_tls_src_files	n/a	string	""	no
consul_version	Consul version number to install	string	"1.9.4"	no
create	Boolean to make module or not	bool	true	no
create_security_group	n/a	bool	true	no
default_telemetry_enabled	n/a	bool	true	no
deployed_networks	n/a	string	""	no
desired_capacity	The desired capacity of asg	string	1	no
enable_scaling	Bool to enable scaling policy	bool	true	no
envoy_enabled	Configure Envoy proxy for Consul Connect	bool	false	no
hardening_enabled	Runs a series of linux hardening playbooks - ansible-collection-hardening	bool	false	no
health_check_enabled	Bool to enable client health check agent	bool	false	no
health_check_grace_period	Time (in seconds) after instance comes into service before checking health	number	60	no
health_check_port	Port number for the health check	string	"5500"	no
iam_instance_profile	The instance profile to associate with the asg - leasve blank to create one regionally scoped.	string	""	no
id	The id to give the ami	string	""	no
instance_count	n/a	string	""	no
instance_storage_driver_type	Type of instance storage (i.e. standard, nitro, raid).	string	"nitro"	no
instance_type	n/a	string	""	no
instance_warmup_time	The time in seconds the instance is estimated to require to warm up	number	9000	no
key_name	The name of the preexisting key to be used instead of the local public_key_path	string	""	no
logging_filter	String for polkadot logging filter	string	"sync=trace,afg=trace,babe=debug"	no
lt_name	The name to give the launch template - defaults to 'name'	string	""	no
max_size	The max size of asg	string	1	no
min_size	The min size of asg	string	1	no
mixed_instances_on_demand_base_capacity	Number of on demand instances to reserve for base capacity	number	0	no
mixed_instances_on_demand_percentage_above_base_capacity	Percentage of on demand instances allowable above base capacity	number	0	no
module_path	n/a	string	""	no
name	The name to give the ASG and associated resources	string	""	no
network_name	Which Polkadot chain to join	string	"kusama"	no
network_settings	n/a	string	""	no
network_stub	The stub name of the Polkadot chain (polkadot = polkadot, kusama = ksmcc3)	string	"ksmcc3"	no
node_exporter_binary_checksum	n/a	string	""	no
node_exporter_binary_url	n/a	string	""	no
node_exporter_enabled	Bool to use when node exporter is enabled	bool	false	no
node_exporter_hash	SHA256 hash of Node Exporter binary	string	"b2503fd932f85f4e5baf161268854bf5d22001869b84f00fd2d1f57b51b72424"	no
node_exporter_password	Password for node exporter	string	"node_exporter_password"	no
node_exporter_url	URL to Node Exporter binary	string	"https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz"	no
node_exporter_user	User for node exporter	string	"node_exporter_user"	no
num_instances	Number of instances for ASG	number	1	no
packer_build_role_arn	The role arn the packer build should use to build the image.	string	""	no
polkadot_additional_common_flags	Additonal common flags for substrate	string	""	no
polkadot_additional_validator_flags	Additonal common flags for validator	string	""	no
polkadot_binary_checksum	n/a	string	""	no
polkadot_binary_url	n/a	string	""	no
polkadot_client_hash	SHA256 hash of Polkadot client binary	string	"cc3bb44e3edc482111fc04b1426b7e2428e8c4b65cf1423c9d892ba97b6f7915"	no
polkadot_client_url	URL to Polkadot client binary	string	"https://github.com/paritytech/polkadot/releases/download/v0.9.16/polkadot"	no
polkadot_prometheus_port	Port number for the Prometheus Metrics exporter built into the Polkadot client	string	"9610"	no
polkadot_restart_enabled	n/a	string	""	no
project	Name of the project for node name	string	"project"	no
prometheus_enabled	Bool to use when Prometheus is enabled	bool	false	no
public_key	The public ssh key	string	""	no
public_key_path	A path to the public key	string	""	no
public_security_group_ports	If create_security_group enabled, and no network_settings blob is supplied, a list of ports to open.	list(string)	[ "30333", "51820" ]	no
retry_join	n/a	string	""	no
role_arn	n/a	string	""	no
root_volume_size	Size in GB for root volume	string	"256"	no
rpc_api_port	Port number for the JSON RPC API	string	"9933"	no
rpc_envoy_port	Port number for the JSON RPC Envoy proxy	string	"21000"	no
scaling_cpu_utilization	The percent CPU utilization for scaling.	number	80	no
security_group_cidr_blocks	If create_security_group enabled, incoming cidr blocks.	list(string)	[ "0.0.0.0/0" ]	no
security_group_ids	n/a	string	""	no
security_groups	The ids of the security groups. Blank to create SG.	list(string)	[]	no
skip_health_check	Bool to skip the health check and give requests while syncing	bool	false	no
source_of_truth_enabled	Bool to use when SOT is enabled	bool	false	no
spot_price	n/a	string	null	no
ssh_user	Username for SSH	string	"ubuntu"	no
subnet_id	n/a	string	""	no
subnet_ids	The ids of the subnets to deploy into	list(string)	null	no
sync_aws_access_key_id	AWS access key ID for SoT sync	string	""	no
sync_aws_secret_access_key	AWS access key for SoT sync	string	""	no
sync_bucket_arn	S3 bucket arn for SoT sync	string	""	no
sync_bucket_kms_key_arn	KMS key used to decrypt S3 bucket for SoT sync	string	""	no
sync_bucket_name	S3 bucket URI for SoT sync	string	""	no
tags	Tags to give resource	map(string)	{}	no
telemetry_url	WSS URL for telemetry	string	""	no
this_skip_health_check	n/a	string	""	no
use_lb	Bool to enable use of load balancer	bool	true	no
use_mixed_instances_policy	Boolean to set if using mixed instance policy	bool	false	no
use_source_of_truth	n/a	string	""	no
vpc_id	n/a	string	""	no
wait_for_capacity_timeout	A maximum duration that Terraform should wait for ASG instances to be healthy before timing out. (See also Waiting for Capacity below.) Setting this to '0' causes Terraform to skip all Capacity Waiting behavior.	string	"10m"	no
wss_api_port	Port number for the Websockets API	string	"9944"	no
wss_envoy_port	Port number for the Websockets Envoy proxy	string	"21001"	no

Outputs

Name	Description
autoscaling_group_arn	n/a
autoscaling_group_id	n/a
autoscaling_group_name	n/a
dns_name	n/a
endpoints_map	n/a
id	n/a
lb_arn	n/a
lb_id	n/a
lb_rpc_target_group_arn	n/a
lb_rpc_target_group_id	n/a
lb_wss_target_group_arn	n/a
lb_wss_target_group_id	n/a
name	n/a
public_ips	n/a
tags	n/a
this_security_group_id	n/a
user_data	n/a

Testing

This module has been packaged with terratest tests

To run them:

1. Install Go
2. Run make test-init from the root of this repo
3. Run make test again from root

License

Apache 2 Licensed. See LICENSE for full details.