Implement AWS Load Balancer Controller addon in a separated file outs…

…ide of blueprints (#179)

Co-authored-by: Soham Kalghatgi <[email protected]>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

CHANGELOG.md

@@ -8,6 +8,7 @@
 - replaced terraform-aws-eks-blueprint addon ebs-csi-driver with module `modules/k8s_eks_addons/ebs-csi.tf`
 - replaced terraform-aws-eks-blueprint addon vpc-cni with module `modules/k8s_eks_addons/vpc-cni.tf`
 - added addon aws-mountpoint-s3-csi-driver with module `modules/k8s_eks_addons/s3-csi.tf`
+- replaced terraform-aws-eks-blueprint addon aws-load-balancer-controller with module `modules/k8s_eks_addons/aws-load-balancer-controller.tf`
 
 ## v0.2.0
 

MAINTENANCE.md

@@ -253,3 +253,42 @@ terraform apply
 ```
 4. Remove `move.tf` file
 
+
+# Migrate aws_load_balancer_controller addon to the module
+To migrate from terraform-aws-eks-blueprint addon aws_load_balancer_controller to custom module `modules/k8s_eks_addons/aws-load-balancer-controller.tf` follow steps:
+
+1. create 'move.tf' in repository root
+2. Add following code:
+```
+moved {
+  from = module.eks-addons.module.aws_load_balancer_controller[0].data.aws_iam_policy_document.aws_load_balancer_controller
+  to   = module.k8s_eks_addons.data.aws_iam_policy_document.aws_load_balancer_controller[0]
+}
+moved {
+  from = module.eks-addons.module.aws_load_balancer_controller[0].aws_iam_policy.aws_load_balancer_controller
+  to   = module.k8s_eks_addons.aws_iam_policy.aws_load_balancer_controller[0]
+}
+moved {
+  from = module.eks-addons.module.aws_load_balancer_controller[0].module.helm_addon.helm_release.addon[0]
+  to   = module.k8s_eks_addons.helm_release.aws_load_balancer_controller[0]
+}
+moved {
+  from = module.eks-addons.module.aws_load_balancer_controller[0].module.helm_addon.module.irsa[0].aws_iam_role.irsa[0]
+  to   = module.k8s_eks_addons.aws_iam_role.aws_load_balancer_controller[0]
+}
+moved {
+  from = module.eks-addons.module.aws_load_balancer_controller[0].module.helm_addon.module.irsa[0].aws_iam_role_policy_attachment.irsa[0]
+  to   = module.k8s_eks_addons.aws_iam_role_policy_attachment.aws_load_balancer_controller[0]
+}
+moved {
+  from = module.eks-addons.module.aws_load_balancer_controller[0].module.helm_addon.module.irsa[0].kubernetes_service_account_v1.irsa[0]
+  to   = module.k8s_eks_addons.kubernetes_service_account_v1.aws_load_balancer_controller[0]
+}
+
+```
+3. Run command:
+```
+terraform apply
+```
+4. Remove `move.tf` file
+

README.md

@@ -306,17 +306,7 @@ foreach ($bucket in $buckets) {
 }
 ```
 
-The remaining infrastructure resources can be deleted via Terraform.
-Due to a bug, Terraform is not able to properly [plan the removal of resources in the right order](https://github.com/aws-ia/terraform-aws-eks-blueprints/issues/353) which leads to a deadlock.
-To workaround the bug, you need to need to remove the `eks-addons` module at first:
-
-```sh
-terraform destroy -target="module.eks-addons"
-```
-
-:warning: **It is important that you have completed the preceding steps. Otherwise, the following command will not finish completly, leaving you in a deadlock state.**
-
-To delete the remaining resources, run the following command:
+The remaining infrastructure resources can be deleted via Terraform by running the following command.
 
 ```sh
 terraform destroy
@@ -473,7 +463,6 @@ Encryption is enabled at all AWS resources that are created by Terraform:
 | Name | Source | Version |
 |------|--------|---------|
 | <a name="module_eks"></a> [eks](#module\_eks) | git::https://github.com/aws-ia/terraform-aws-eks-blueprints.git | v4.32.1 |
-| <a name="module_eks-addons"></a> [eks-addons](#module\_eks-addons) | git::https://github.com/aws-ia/terraform-aws-eks-blueprints.git//modules/kubernetes-addons | v4.32.1 |
 | <a name="module_k8s_eks_addons"></a> [k8s\_eks\_addons](#module\_k8s\_eks\_addons) | ./modules/k8s_eks_addons | n/a |
 | <a name="module_security_group"></a> [security\_group](#module\_security\_group) | terraform-aws-modules/security-group/aws | ~> 4 |
 | <a name="module_security_group_license_server"></a> [security\_group\_license\_server](#module\_security\_group\_license\_server) | terraform-aws-modules/security-group/aws | ~> 4 |
@@ -552,12 +541,12 @@ Encryption is enabled at all AWS resources that are created by Terraform:
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
+| <a name="input_aws_load_balancer_controller_config"></a> [aws\_load\_balancer\_controller\_config](#input\_aws\_load\_balancer\_controller\_config) | Input configuration for load\_balancer\_controller deployed with helm release. By setting key 'enable' to 'true', load\_balancer\_controller release will be deployed. 'helm\_repository' is an URL for the repository of load\_balancer\_controller helm chart, where 'helm\_version' is its respective version of a chart. 'chart\_values' is used for changing default values.yaml of a load\_balancer\_controller chart. | <pre>object({<br>    enable          = optional(bool, false)<br>    helm_repository = optional(string, "https://aws.github.io/eks-charts")<br>    helm_version    = optional(string, "1.4.5")<br>    chart_values = optional(string, <<-YAML<br><br>    YAML<br>    )<br>  })</pre> | <pre>{<br>  "enable": false<br>}</pre> | no |
 | <a name="input_cloudwatch_retention"></a> [cloudwatch\_retention](#input\_cloudwatch\_retention) | Global cloudwatch retention period for the EKS, VPC, SSM, and PostgreSQL logs. | `number` | `7` | no |
 | <a name="input_cluster_autoscaler_config"></a> [cluster\_autoscaler\_config](#input\_cluster\_autoscaler\_config) | Input configuration for cluster-autoscaler deployed with helm release. By setting key 'enable' to 'true', cluster-autoscaler release will be deployed. 'helm\_repository' is an URL for the repository of cluster-autoscaler helm chart, where 'helm\_version' is its respective version of a chart. 'chart\_values' is used for changing default values.yaml of a cluster-autoscaler chart. | <pre>object({<br>    enable          = optional(bool, true)<br>    helm_repository = optional(string, "https://kubernetes.github.io/autoscaler")<br>    helm_version    = optional(string, "9.37.0")<br>    chart_values = optional(string, <<-YAML<br><br>    YAML<br>    )<br>  })</pre> | `{}` | no |
 | <a name="input_codemeter"></a> [codemeter](#input\_codemeter) | Download link for codemeter rpm package. | `string` | `"https://www.wibu.com/support/user/user-software/file/download/13346.html?tx_wibudownloads_downloadlist%5BdirectDownload%5D=directDownload&tx_wibudownloads_downloadlist%5BuseAwsS3%5D=0&cHash=8dba7ab094dec6267346f04fce2a2bcd"` | no |
 | <a name="input_coredns_config"></a> [coredns\_config](#input\_coredns\_config) | Input configuration for AWS EKS add-on coredns. By setting key 'enable' to 'true', coredns add-on is deployed. Key 'configuration\_values' is used to change add-on configuration. Its content should follow add-on configuration schema (see https://aws.amazon.com/blogs/containers/amazon-eks-add-ons-advanced-configuration/). | <pre>object({<br>    enable               = optional(bool, true)<br>    configuration_values = optional(string, null)<br>  })</pre> | <pre>{<br>  "enable": true<br>}</pre> | no |
 | <a name="input_ecr_pullthrough_cache_rule_config"></a> [ecr\_pullthrough\_cache\_rule\_config](#input\_ecr\_pullthrough\_cache\_rule\_config) | Specifies if ECR pull through cache rule and accompanying resources will be created. Key 'enable' indicates whether pull through cache rule needs to be enabled for the cluster. When 'enable' is set to 'true', key 'exist' indicates whether pull through cache rule already exists for region's private ECR. If key 'enable' is set to 'true', IAM policy will be attached to the cluster's nodes. Additionally, if 'exist' is set to 'false', credentials for upstream registry and pull through cache rule will be created | <pre>object({<br>    enable = bool<br>    exist  = bool<br>  })</pre> | <pre>{<br>  "enable": false,<br>  "exist": false<br>}</pre> | no |
-| <a name="input_enable_aws_for_fluentbit"></a> [enable\_aws\_for\_fluentbit](#input\_enable\_aws\_for\_fluentbit) | Install FluentBit to send container logs to CloudWatch. | `bool` | `false` | no |
 | <a name="input_enable_ivs"></a> [enable\_ivs](#input\_enable\_ivs) | n/a | `bool` | `false` | no |
 | <a name="input_enable_patching"></a> [enable\_patching](#input\_enable\_patching) | Scans license server EC2 instance and EKS nodes for updates. Installs patches on license server automatically. EKS nodes need to be updated manually. | `bool` | `false` | no |
 | <a name="input_gpuNodeCountMax"></a> [gpuNodeCountMax](#input\_gpuNodeCountMax) | The maximum number of nodes for gpu job execution | `number` | `12` | no |

k8s-eks-addons.tf

@@ -1,10 +1,11 @@
 module "k8s_eks_addons" {
   source = "./modules/k8s_eks_addons"
 
-  ingress_nginx_config      = merge(var.ingress_nginx_config, { subnets_ids = local.public_subnets })
-  cluster_autoscaler_config = var.cluster_autoscaler_config
-  coredns_config            = var.coredns_config
-  s3_csi_config             = var.s3_csi_config
+  ingress_nginx_config                = merge(var.ingress_nginx_config, { subnets_ids = local.public_subnets })
+  cluster_autoscaler_config           = var.cluster_autoscaler_config
+  coredns_config                      = var.coredns_config
+  s3_csi_config                       = var.s3_csi_config
+  aws_load_balancer_controller_config = var.aws_load_balancer_controller_config
 
   addon_context = {
     aws_caller_identity_account_id = data.aws_caller_identity.current.account_id
@@ -16,5 +17,5 @@ module "k8s_eks_addons" {
     tags                           = var.tags
   }
 
-  depends_on = [module.eks.eks_cluster_arn]
+  depends_on = [module.eks.eks_cluster_arn, module.vpc]
 }

k8s.tf

@@ -14,26 +14,6 @@ module "eks" {
   managed_node_groups                    = merge(local.default_managed_node_pools, var.gpuNodePool ? local.gpu_node_pool : {}, var.ivsGpuNodePool ? local.ivsgpu_node_pool : {})
 }
 
-
-module "eks-addons" {
-  source                              = "git::https://github.com/aws-ia/terraform-aws-eks-blueprints.git//modules/kubernetes-addons?ref=v4.32.1"
-  eks_cluster_id                      = module.eks.eks_cluster_id
-  enable_aws_load_balancer_controller = false
-  enable_aws_for_fluentbit            = var.enable_aws_for_fluentbit
-  tags                                = var.tags
-
-  aws_for_fluentbit_helm_config = {
-    values = [templatefile("${path.module}/templates/fluentbit_values.yaml", {
-      aws_region           = data.aws_region.current.name,
-      log_group_name       = local.log_group_name,
-      service_account_name = "aws-for-fluent-bit-sa"
-    })]
-    dependency_update = true
-  }
-
-  #depends_on                     = [module.eks.managed_node_groups]
-}
-
 data "aws_eks_node_group" "default" {
   cluster_name    = local.infrastructurename
   node_group_name = replace(module.eks.managed_node_groups[0]["default"]["managed_nodegroup_id"][0], "${local.infrastructurename}:", "")

locals.tf

@@ -13,7 +13,6 @@ locals {
   use_private_subnets_ids                   = length(var.private_subnet_ids) == 0 ? false : true
   use_public_subnet_ids                     = length(var.public_subnet_ids) == 0 ? false : true
   infrastructurename                        = var.infrastructurename
-  log_group_name                            = "/${module.eks.eks_cluster_id}/worker-fluentbit-logs"
   account_id                                = data.aws_caller_identity.current.account_id
   region                                    = data.aws_region.current.name
   license_server_role                       = "${local.infrastructurename}-license-server-role"

modules/k8s_eks_addons/ingress-nginx.tf

@@ -18,9 +18,13 @@ resource "helm_release" "ingress_nginx" {
   dependency_update = true
   values = [
     templatefile("${path.module}/templates/nginx_values.yaml", {
-      public_subnets = join(", ", var.ingress_nginx_config.subnets_ids)
+      public_subnets         = join(", ", var.ingress_nginx_config.subnets_ids)
+      protocol               = var.aws_load_balancer_controller_config.enable ? "ssl" : "tcp"
+      aws_load_balancer_type = var.aws_load_balancer_controller_config.enable ? "external" : "nlb"
+      aws_load_target-type   = var.aws_load_balancer_controller_config.enable ? "service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip" : ""
     }),
     var.ingress_nginx_config.chart_values
   ]
-  timeout = 1200
+  timeout    = 1200
+  depends_on = [helm_release.aws_load_balancer_controller]
 }