Skip to content

Commit

Permalink
separate module for serverless
Browse files Browse the repository at this point in the history
  • Loading branch information
magreenbaum committed May 14, 2024
1 parent 6d91838 commit d0a0684
Show file tree
Hide file tree
Showing 9 changed files with 179 additions and 81 deletions.
5 changes: 0 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,6 @@ No modules.
| [aws_msk_cluster_policy.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/msk_cluster_policy) | resource |
| [aws_msk_configuration.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/msk_configuration) | resource |
| [aws_msk_scram_secret_association.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/msk_scram_secret_association) | resource |
| [aws_msk_serverless_cluster.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/msk_serverless_cluster) | resource |
| [aws_msk_vpc_connection.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/msk_vpc_connection) | resource |
| [aws_mskconnect_custom_plugin.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/mskconnect_custom_plugin) | resource |
| [aws_mskconnect_worker_configuration.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/mskconnect_worker_configuration) | resource |
Expand Down Expand Up @@ -197,7 +196,6 @@ No modules.
| <a name="input_create_connect_worker_configuration"></a> [create\_connect\_worker\_configuration](#input\_create\_connect\_worker\_configuration) | Determines whether to create connect worker configuration | `bool` | `false` | no |
| <a name="input_create_schema_registry"></a> [create\_schema\_registry](#input\_create\_schema\_registry) | Determines whether to create a Glue schema registry for managing Avro schemas for the cluster | `bool` | `true` | no |
| <a name="input_create_scram_secret_association"></a> [create\_scram\_secret\_association](#input\_create\_scram\_secret\_association) | Determines whether to create SASL/SCRAM secret association | `bool` | `false` | no |
| <a name="input_create_serverless_cluster"></a> [create\_serverless\_cluster](#input\_create\_serverless\_cluster) | Determine whether to create a serverless cluster | `bool` | `false` | no |
| <a name="input_enable_storage_autoscaling"></a> [enable\_storage\_autoscaling](#input\_enable\_storage\_autoscaling) | Determines whether autoscaling is enabled for storage | `bool` | `true` | no |
| <a name="input_encryption_at_rest_kms_key_arn"></a> [encryption\_at\_rest\_kms\_key\_arn](#input\_encryption\_at\_rest\_kms\_key\_arn) | You may specify a KMS key short ID or ARN (it will always output an ARN) to use for encrypting your data at rest. If no key is specified, an AWS managed KMS ('aws/msk' managed service) key will be used for encrypting the data at rest | `string` | `null` | no |
| <a name="input_encryption_in_transit_client_broker"></a> [encryption\_in\_transit\_client\_broker](#input\_encryption\_in\_transit\_client\_broker) | Encryption setting for data in transit between clients and brokers. Valid values: `TLS`, `TLS_PLAINTEXT`, and `PLAINTEXT`. Default value is `TLS` | `string` | `null` | no |
Expand All @@ -219,7 +217,6 @@ No modules.
| <a name="input_schema_registries"></a> [schema\_registries](#input\_schema\_registries) | A map of schema registries to be created | `map(any)` | `{}` | no |
| <a name="input_schemas"></a> [schemas](#input\_schemas) | A map schemas to be created within the schema registry | `map(any)` | `{}` | no |
| <a name="input_scram_secret_association_secret_arn_list"></a> [scram\_secret\_association\_secret\_arn\_list](#input\_scram\_secret\_association\_secret\_arn\_list) | List of AWS Secrets Manager secret ARNs to associate with SCRAM | `list(string)` | `[]` | no |
| <a name="input_serverless_vpc_config"></a> [serverless\_vpc\_config](#input\_serverless\_vpc\_config) | Serverless cluster VPC configurations | `any` | `{}` | no |
| <a name="input_storage_mode"></a> [storage\_mode](#input\_storage\_mode) | Controls storage mode for supported storage tiers. Valid values are: `LOCAL` or `TIERED` | `string` | `null` | no |
| <a name="input_tags"></a> [tags](#input\_tags) | A map of tags to assign to the resources created | `map(string)` | `{}` | no |
| <a name="input_timeouts"></a> [timeouts](#input\_timeouts) | Create, update, and delete timeout configurations for the cluster | `map(string)` | `{}` | no |
Expand Down Expand Up @@ -249,8 +246,6 @@ No modules.
| <a name="output_schema_registries"></a> [schema\_registries](#output\_schema\_registries) | A map of output attributes for the schema registries created |
| <a name="output_schemas"></a> [schemas](#output\_schemas) | A map of output attributes for the schemas created |
| <a name="output_scram_secret_association_id"></a> [scram\_secret\_association\_id](#output\_scram\_secret\_association\_id) | Amazon Resource Name (ARN) of the MSK cluster |
| <a name="output_serverless_arn"></a> [serverless\_arn](#output\_serverless\_arn) | The ARN of the serverless cluster |
| <a name="output_serverless_cluster_uuid"></a> [serverless\_cluster\_uuid](#output\_serverless\_cluster\_uuid) | UUID of the serverless cluster, for use in IAM policies |
| <a name="output_vpc_connections"></a> [vpc\_connections](#output\_vpc\_connections) | A map of output attributes for the VPC connections created |
| <a name="output_zookeeper_connect_string"></a> [zookeeper\_connect\_string](#output\_zookeeper\_connect\_string) | A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster. The returned values are sorted alphabetically |
| <a name="output_zookeeper_connect_string_tls"></a> [zookeeper\_connect\_string\_tls](#output\_zookeeper\_connect\_string\_tls) | A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster via TLS. The returned values are sorted alphabetically |
Expand Down
11 changes: 4 additions & 7 deletions examples/serverless/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -19,15 +19,12 @@ locals {
}

module "msk_serverless_cluster" {
source = "../.."
source = "../../modules/serverless"

name = local.name
create_serverless_cluster = true
name = local.name

serverless_vpc_config = {
security_group_ids = [module.security_group.security_group_id]
subnet_ids = module.vpc.private_subnets
}
security_group_ids = [module.security_group.security_group_id]
subnet_ids = module.vpc.private_subnets

create_cluster_policy = true
cluster_policy_statements = {
Expand Down
49 changes: 10 additions & 39 deletions main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
################################################################################

resource "aws_msk_cluster" "this" {
count = var.create && !var.create_serverless_cluster ? 1 : 0
count = var.create ? 1 : 0

broker_node_group_info {
az_distribution = var.broker_node_az_distribution
Expand Down Expand Up @@ -173,7 +173,7 @@ resource "aws_msk_cluster" "this" {
################################################################################

resource "aws_msk_vpc_connection" "this" {
for_each = { for k, v in var.vpc_connections : k => v if var.create && !var.create_serverless_cluster }
for_each = { for k, v in var.vpc_connections : k => v if var.create }

authentication = each.value.authentication
client_subnets = each.value.client_subnets
Expand All @@ -191,7 +191,7 @@ resource "aws_msk_vpc_connection" "this" {
resource "aws_msk_cluster_policy" "this" {
count = var.create && var.create_cluster_policy ? 1 : 0

cluster_arn = !var.create_serverless_cluster ? aws_msk_cluster.this[0].arn : aws_msk_serverless_cluster.this[0].arn
cluster_arn = aws_msk_cluster.this[0].arn
policy = data.aws_iam_policy_document.this[0].json
}

Expand Down Expand Up @@ -248,13 +248,13 @@ data "aws_iam_policy_document" "this" {
################################################################################

resource "random_id" "this" {
count = var.create && var.create_configuration && !var.create_serverless_cluster ? 1 : 0
count = var.create && var.create_configuration ? 1 : 0

byte_length = 8
}

resource "aws_msk_configuration" "this" {
count = var.create && var.create_configuration && !var.create_serverless_cluster ? 1 : 0
count = var.create && var.create_configuration ? 1 : 0

name = format("%s-%s", coalesce(var.configuration_name, var.name), random_id.this[0].dec)
description = var.configuration_description
Expand All @@ -271,7 +271,7 @@ resource "aws_msk_configuration" "this" {
################################################################################

resource "aws_msk_scram_secret_association" "this" {
count = var.create && var.create_scram_secret_association && try(var.client_authentication.sasl.scram, false) && !var.create_serverless_cluster ? 1 : 0
count = var.create && var.create_scram_secret_association && try(var.client_authentication.sasl.scram, false) ? 1 : 0

cluster_arn = aws_msk_cluster.this[0].arn
secret_arn_list = var.scram_secret_association_secret_arn_list
Expand All @@ -282,11 +282,11 @@ resource "aws_msk_scram_secret_association" "this" {
################################################################################

locals {
cloudwatch_log_group = var.create && var.create_cloudwatch_log_group && !var.create_serverless_cluster ? aws_cloudwatch_log_group.this[0].name : var.cloudwatch_log_group_name
cloudwatch_log_group = var.create && var.create_cloudwatch_log_group ? aws_cloudwatch_log_group.this[0].name : var.cloudwatch_log_group_name
}

resource "aws_cloudwatch_log_group" "this" {
count = var.create && var.create_cloudwatch_log_group && !var.create_serverless_cluster ? 1 : 0
count = var.create && var.create_cloudwatch_log_group ? 1 : 0

name = coalesce(var.cloudwatch_log_group_name, "/aws/msk/${var.name}")
retention_in_days = var.cloudwatch_log_group_retention_in_days
Expand All @@ -300,7 +300,7 @@ resource "aws_cloudwatch_log_group" "this" {
################################################################################

resource "aws_appautoscaling_target" "this" {
count = var.create && var.enable_storage_autoscaling && !var.create_serverless_cluster ? 1 : 0
count = var.create && var.enable_storage_autoscaling ? 1 : 0

max_capacity = var.scaling_max_capacity
min_capacity = 1
Expand All @@ -311,7 +311,7 @@ resource "aws_appautoscaling_target" "this" {
}

resource "aws_appautoscaling_policy" "this" {
count = var.create && var.enable_storage_autoscaling && !var.create_serverless_cluster ? 1 : 0
count = var.create && var.enable_storage_autoscaling ? 1 : 0

name = "${var.name}-broker-storage-scaling"
policy_type = "TargetTrackingScaling"
Expand Down Expand Up @@ -353,32 +353,3 @@ resource "aws_glue_schema" "this" {

tags = merge(var.tags, try(each.value.tags, {}))
}

################################################################################
# Serverless Cluster
################################################################################

resource "aws_msk_serverless_cluster" "this" {
count = var.create && var.create_serverless_cluster ? 1 : 0

client_authentication {
sasl {
iam {
enabled = true
}
}
}

cluster_name = var.name

dynamic "vpc_config" {
for_each = try([var.serverless_vpc_config], [])

content {
security_group_ids = try(vpc_config.value.security_group_ids, null)
subnet_ids = vpc_config.value.subnet_ids
}
}

tags = var.tags
}
82 changes: 82 additions & 0 deletions modules/serverless/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
################################################################################
# Serverless Cluster
################################################################################
resource "aws_msk_serverless_cluster" "this" {
count = var.create ? 1 : 0

client_authentication {
sasl {
iam {
enabled = true
}
}
}

cluster_name = var.name

vpc_config {
security_group_ids = var.security_group_ids
subnet_ids = var.subnet_ids
}

tags = var.tags
}

################################################################################
# Cluster Policy
################################################################################

resource "aws_msk_cluster_policy" "this" {
count = var.create && var.create_cluster_policy ? 1 : 0

cluster_arn = aws_msk_serverless_cluster.this[0].arn
policy = data.aws_iam_policy_document.this[0].json
}

data "aws_iam_policy_document" "this" {
count = var.create && var.create_cluster_policy ? 1 : 0

source_policy_documents = var.cluster_source_policy_documents
override_policy_documents = var.cluster_override_policy_documents

dynamic "statement" {
for_each = var.cluster_policy_statements

content {
sid = try(statement.value.sid, null)
actions = try(statement.value.actions, null)
not_actions = try(statement.value.not_actions, null)
effect = try(statement.value.effect, null)
resources = try(statement.value.resources, [aws_msk_serverless_cluster.this[0].arn])
not_resources = try(statement.value.not_resources, null)

dynamic "principals" {
for_each = try(statement.value.principals, [])

content {
type = principals.value.type
identifiers = principals.value.identifiers
}
}

dynamic "not_principals" {
for_each = try(statement.value.not_principals, [])

content {
type = not_principals.value.type
identifiers = not_principals.value.identifiers
}
}

dynamic "condition" {
for_each = try(statement.value.conditions, [])

content {
test = condition.value.test
values = condition.value.values
variable = condition.value.variable
}
}
}
}
}
13 changes: 13 additions & 0 deletions modules/serverless/outputs.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
################################################################################
# Serverless Cluster
################################################################################

output "serverless_arn" {
description = "The ARN of the serverless cluster"
value = try(aws_msk_serverless_cluster.this[0].arn, null)
}

output "serverless_cluster_uuid" {
description = "UUID of the serverless cluster, for use in IAM policies"
value = try(aws_msk_serverless_cluster.this[0].cluster_uuid, null)
}
60 changes: 60 additions & 0 deletions modules/serverless/variables.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
################################################################################
# Serverless Cluster
################################################################################
variable "create" {
description = "Determines whether cluster resources will be created"
type = bool
default = true
}

variable "name" {
description = "Name of the MSK serverless cluster"
type = string
default = null
}

variable "security_group_ids" {
description = "Specifies up to five security groups that control inbound and outbound traffic for the serverless cluster"
type = list(string)
default = null
}

variable "subnet_ids" {
description = "A list of subnets in at least two different Availability Zones that host your client applications"
type = list(string)
default = null
}

variable "tags" {
description = "A map of tags to assign to the resources created"
type = map(string)
default = {}
}

################################################################################
# Cluster Policy
################################################################################

variable "create_cluster_policy" {
description = "Determines whether to create an MSK cluster policy"
type = bool
default = false
}

variable "cluster_source_policy_documents" {
description = "Source policy documents for cluster policy"
type = list(string)
default = null
}

variable "cluster_override_policy_documents" {
description = "Override policy documents for cluster policy"
type = list(string)
default = null
}

variable "cluster_policy_statements" {
description = "Map of policy statements for cluster policy"
type = any
default = null
}
10 changes: 10 additions & 0 deletions modules/serverless/versions.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
terraform {
required_version = ">= 1.0"

required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.21"
}
}
}
14 changes: 0 additions & 14 deletions outputs.tf
Original file line number Diff line number Diff line change
Expand Up @@ -153,17 +153,3 @@ output "connect_worker_configuration_latest_revision" {
description = "An ID of the latest successfully created revision of the worker configuration"
value = try(aws_mskconnect_worker_configuration.this[0].latest_revision, null)
}

################################################################################
# Serverless Cluster
################################################################################

output "serverless_arn" {
description = "The ARN of the serverless cluster"
value = try(aws_msk_serverless_cluster.this[0].arn, null)
}

output "serverless_cluster_uuid" {
description = "UUID of the serverless cluster, for use in IAM policies"
value = try(aws_msk_serverless_cluster.this[0].cluster_uuid, null)
}
16 changes: 0 additions & 16 deletions variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -375,19 +375,3 @@ variable "connect_worker_config_properties_file_content" {
type = string
default = null
}

################################################################################
# Serverless Cluster
################################################################################

variable "create_serverless_cluster" {
description = "Determine whether to create a serverless cluster"
type = bool
default = false
}

variable "serverless_vpc_config" {
description = "Serverless cluster VPC configurations"
type = any
default = {}
}

0 comments on commit d0a0684

Please sign in to comment.