Skip to content

Latest commit

 

History

History
350 lines (302 loc) · 34.2 KB

README.md

File metadata and controls

350 lines (302 loc) · 34.2 KB
Raw

EKS

Prerequisites

Resources Provisioned

  • EKS Cluster
  • KMS Key is used to encrypt K8S Secrets
  • IAM Role for Service Account is enabled
  • KMS Key for EBS volumes is created and ASG is given permission to use the key
  • A "default" self managed node group on Bottlerocket is provisioned.

In addition, it is expected that all worker nodes will share one IAM Role and a common security group.

  • IAM Role contains the necessary policies for nodes to join the EKS cluster and optionally manage ENI for CNI purposes. IAM Role for Service Account usage is strongly recommended.
  • The Security Group has default rules to allow the cluster to function. Additional groups can be added for additional node groups.

Instance refresh on ASG is handled by AWS Node Termination Handler. For the purposes of instance refresh, the following resources are created:

  • SQS Queue where events published to be consumed by Node Termination Handler is published.

The Queue ARN will be subsequently used by the eks_self_managed_nodes module to provision additional node groups.

Usage

Defining Providers

Definining providers in reusable modules is deprecated and causes features like modules for_each and count to not work. In addition to the aws providers, the main module and submodules can require additional Kubernetes providers to be configured.

Main Module

The main module uses the kubernetes provider.

provider "aws" {
  # ...
}

module "eks" {
  # ...
  create_aws_auth_configmap = false
  manage_aws_auth_configmap = false # set this to true after cluster creation is completed and reapply to set IRSA
}

data "aws_eks_cluster_auth" "this" {
  name = module.eks.cluster_name
}

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  token                  = data.aws_eks_cluster_auth.this.token
}

Sub-modules

Other modules might make use of the kubernetes or helm providers

provider "aws" {
  # ...
}

data "aws_eks_cluster" "this" {
  name = var.cluster_name
}

data "aws_eks_cluster_auth" "this" {
  name = var.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.this.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.this.token
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.this.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.this.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.this.token
  }

  experiments {
    manifest = true
  }
}

Karpenter

We need to allow Karpenter controller to start EC2 instances hence, we need to add role mapping for it:

role_mapping = local.autoscaling_mode == "karpenter" ? concat([
    for role in local.eks_master_roles :
    {
      rolearn  = role.arn
      groups   = ["system:masters"]
      username = role.user
    }],
    [
      {
        rolearn = module.eks.worker_iam_role_arn
        groups = [
          "system:bootstrappers",
          "system:nodes",
        ]
        username = "system:node:{{EC2PrivateDNSName}}"
      }
    ]
    ) : [
    for role in local.eks_master_roles :
    {
      rolearn  = role.arn
      groups   = ["system:masters"]
      username = role.user
    }
  ]

Then we create Karpenter module with configuration as follows:

locals {
  # Karpenter Provisioners Config
  karpenter_provisioners = [
    {
      name                           = "provisioner_name"
      provider_ref_nodetemplate_name = "default"
      karpenter_provisioner_node_labels = {
        "label_key" = "label_value"
      }

      karpenter_provisioner_node_taints = [
        {
          key       = "taintkey",
          value     = "taintvalue",
          effect    = "NoSchedule"
          timeAdded = timestamp() # required if not terraform plan complains
        }
      ]
    },
  ]
  # Karpenter Nodetemplate Config
  karpenter_nodetemplates = [
    {
      name = "default"
      karpenter_subnet_selector_map = {
        "Name" = "subnet-name-here-*"
      }
      karpenter_security_group_selector_map = {
        "aws-ids" = module.eks.worker_security_group_id
      }
      karpenter_nodetemplate_tag_map = {
        "karpenter.sh/discovery" = module.eks.cluster_name
      }
    },
  ]
}

module "karpenter" {
  # source  = "SPHTech-Platform/eks/aws//modules/essentials"
  # version = "~> 0.8.0"

  source = "git::https://github.com/SPHTech-Platform/terraform-aws-eks.git//modules/karpenter?ref=karpenter"

  cluster_name        = local.cluster_name
  cluster_endpoint    = data.aws_eks_cluster.this.endpoint
  oidc_provider_arn   = module.eks.oidc_provider_arn
  worker_iam_role_arn = module.eks.worker_iam_role_arn

  autoscaling_mode        = local.autoscaling_mode
  karpenter_provisioners  = local.karpenter_provisioners
  karpenter_nodetemplates = local.karpenter_nodetemplates

}

Requirements

Name Version
terraform >= 1.4
aws >= 5.70
helm >= 2.16
kubectl >= 2.1
kubernetes >= 2.33

Providers

Name Version
aws >= 5.70
kubernetes >= 2.33

Modules

Name Source Version
aws_ebs_csi_pod_identity terraform-aws-modules/eks-pod-identity/aws ~> 1.5.0
aws_vpc_cni_pod_identity terraform-aws-modules/eks-pod-identity/aws ~> 1.5.0
ebs_csi_irsa_role terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks ~> 5.47
eks terraform-aws-modules/eks/aws ~> 20.29.0
eks_aws_auth terraform-aws-modules/eks/aws//modules/aws-auth ~> 20.29.0
fargate_profiles ./modules/fargate_profile n/a
karpenter ./modules/karpenter n/a
kms_ebs SPHTech-Platform/kms/aws ~> 0.2.0
kms_secret SPHTech-Platform/kms/aws ~> 0.2.0
node_groups ./modules/eks_managed_nodes n/a
vpc_cni_irsa_role terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks ~> 5.47

Resources

Name Type
aws_iam_role.cluster resource
aws_iam_role.workers resource
aws_iam_role_policy.ebs_csi_kms resource
aws_iam_role_policy_attachment.cluster resource
aws_iam_role_policy_attachment.workers resource
aws_iam_service_linked_role.autoscaling resource
kubernetes_config_map_v1.amazon_vpc_cni resource
kubernetes_manifest.fargate_node_security_group_policy resource
kubernetes_manifest.fargate_node_security_group_policy_for_karpenter resource
aws_ami.eks_default_bottlerocket data source
aws_arn.cluster data source
aws_caller_identity.current data source
aws_iam_policy_document.ec2_assume_role_policy data source
aws_iam_policy_document.eks_assume_role_policy data source
aws_iam_policy_document.kms_csi_ebs data source
aws_iam_policy_document.kms_ebs data source
aws_partition.current data source
aws_subnet.subnets data source

Inputs

Name Description Type Default Required
access_entries Map of access entries to add to the cluster any {} no
additional_karpenter_security_group_ids Additional security group IDs to add to the Karpenter node groups list(string) [] no
authentication_mode The authentication mode for the cluster. Valid values are CONFIG_MAP, API or API_AND_CONFIG_MAP string "API" no
autoscaling_mode Autoscaling mode: cluster_autoscaler or karpenter string "karpenter" no
aws_auth_fargate_profile_pod_execution_role_arns List of Fargate profile pod execution role ARNs to add to the aws-auth configmap list(string) [] no
cloudwatch_log_group_tags A map of additional tags to add to the cloudwatch log group created map(string) {} no
cluster_additional_security_group_ids List of additional, externally created security group IDs to attach to the cluster control plane list(string) [] no
cluster_addons Map of cluster addon configurations to enable for the cluster. Addon name can be the map keys or set with name any {} no
cluster_addons_timeouts Create, update, and delete timeout configurations for the cluster addons map(string) {} no
cluster_enabled_log_types A list of the desired control plane logs to enable. For more information, see Amazon EKS Control Plane Logging documentation (https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html) list(string) 
[
  "audit",
  "api",
  "authenticator"
]
 no
cluster_endpoint_private_access Indicates whether or not the Amazon EKS private API server endpoint is enabled bool true no
cluster_endpoint_public_access Indicates whether or not the Amazon EKS public API server endpoint is enabled bool true no
cluster_endpoint_public_access_cidrs List of CIDR blocks which can access the Amazon EKS public API server endpoint list(string) 
[
  "0.0.0.0/0"
]
 no
cluster_iam_boundary IAM boundary for the cluster IAM role, if any string null no
cluster_iam_role Cluster IAM Role name. If undefined, is the same as the cluster name string "" no
cluster_ip_family The IP family used to assign Kubernetes pod and service addresses. Valid values are ipv4 (default) and ipv6. You can only specify an IP family when you create a cluster, changing this value will force a new cluster to be created string "ipv4" no
cluster_name EKS Cluster Name string n/a yes
cluster_security_group_additional_rules List of additional security group rules to add to the cluster security group created. Set source_node_security_group = true inside rules to set the node_security_group as source any {} no
cluster_security_group_name Cluster security group name string null no
cluster_service_ipv4_cidr The CIDR block to assign Kubernetes service IP addresses from. If you don't specify a block, Kubernetes assigns addresses from either the 10.100.0.0/16 or 172.20.0.0/16 CIDR blocks string null no
cluster_service_ipv6_cidr The CIDR block to assign Kubernetes pod and service IP addresses from if ipv6 was specified when the cluster was created. Kubernetes assigns service addresses from the unique local address range (fc00::/7) because you can't specify a custom IPv6 CIDR block when you create the cluster string null no
cluster_version EKS Cluster Version string "1.30" no
create_aws_auth_configmap Determines whether to create the aws-auth configmap. NOTE - this is only intended for scenarios where the configmap does not exist (i.e. - when using only self-managed node groups). Most users should use manage_aws_auth_configmap bool false no
create_aws_observability_ns Whether to create AWS Observability Namespace. bool true no
create_aws_observability_ns_for_karpenter Create aws-observability namespace flag bool false no
create_cluster_security_group Determines if a security group is created for the cluster. Note: the EKS service creates a primary security group for the cluster by default bool true no
create_cni_ipv6_iam_policy Whether to create CNI IPv6 IAM policy. bool false no
create_fargate_log_group_for_karpenter value for create_fargate_log_group bool false no
create_fargate_logger_configmap Whether to create AWS Fargate logger configmap. bool true no
create_fargate_logger_configmap_for_karpenter create_fargate_logger_configmap flag bool false no
create_fargate_logging_policy_for_karpenter value for create_fargate_logging_policy bool false no
create_fargate_profile_for_karpenter Create fargate profile flag bool false no
create_node_security_group Determines whether to create a security group for the node groups or use the existing node_security_group_id bool true no
default_group_ami_id The AMI from which to launch the defualt group instance. If not supplied, EKS will use its own default image string "" no
default_group_instance_types Instance type for the default node group list(string) 
[
  "m5a.xlarge",
  "m5.xlarge",
  "m5n.xlarge",
  "m5zn.xlarge"
]
 no
default_group_launch_template_name Name of the default node group launch template string "default" no
default_group_max_size Configuration for max default node group size number 5 no
default_group_min_size Configuration for min default node group size number 1 no
default_group_name Name of the default node group string "default" no
default_group_node_labels Additional node label for default group map(string) {} no
default_group_subnet_ids Subnet IDs to create the default group ASGs in list(string) [] no
default_group_volume_size Size of the persistentence volume for the default group number 50 no
eks_managed_node_group_defaults Map of EKS managed node group default configurations any 
{
  "create_iam_role": false,
  "disk_size": 50,
  "ebs_optimized": true,
  "enable_monitoring": true,
  "metadata_options": {
    "http_endpoint": "enabled",
    "http_protocol_ipv6": "disabled",
    "http_put_response_hop_limit": 1,
    "http_tokens": "required",
    "instance_metadata_tags": "disabled"
  },
  "protect_from_scale_in": false,
  "update_launch_template_default_version": true
}
 no
eks_managed_node_groups Map of EKS managed node group definitions to create any {} no
enable_cluster_creator_admin_permissions Indicates whether or not to add the cluster creator (the identity used by Terraform) as an administrator via access entry bool true no
enable_cluster_windows_support Determines whether to create the amazon-vpc-cni configmap and windows worker roles into aws-auth. bool false no
enable_pod_identity_for_eks_addons Enable pod identity for eks addons, Note - Default is false because AWS Terraform Provider still DOESN'T Support Pod Identity Association for EKS Addons bool false no
enable_pod_identity_for_karpenter Enable pod identity for karpenter bool false no
enable_v1_permissions_for_karpenter Determines whether to enable permissions suitable for v1+ (true) or for v0.33.x-v0.37.x (false) bool true no
fargate_cluster Whether to create eks cluster with fargate mode. If true, default node group also will be fargate, otherwise managed bool false no
fargate_profile_defaults Map of Fargate Profile default configurations any {} no
fargate_profiles Map of maps of fargate_profiles to create any {} no
force_imdsv2 Force IMDSv2 metadata server. bool true no
force_irsa Force usage of IAM Roles for Service Account bool true no
iam_role_additional_policies Additional policies to be added to the IAM role set(string) [] no
karpenter_chart_version Chart version for Karpenter string "1.0.8" no
karpenter_crd_chart_version Chart version for Karpenter CRDs same version as karpenter_chart_version string "1.0.8" no
karpenter_default_subnet_selector_tags Subnet selector tags for Karpenter default node class map(string) 
{
  "kubernetes.io/role/internal-elb": "1"
}
 no
karpenter_nodeclasses List of nodetemplate maps 
list(object({
    nodeclass_name                         = string
    karpenter_subnet_selector_maps         = list(map(any))
    karpenter_security_group_selector_maps = list(map(any))
    karpenter_ami_selector_maps            = list(map(any))
    karpenter_node_role                    = string
    karpenter_node_tags_map                = map(string)
    karpenter_ami_family                   = string
    karpenter_node_user_data               = string
    karpenter_node_metadata_options        = map(any)
    karpenter_block_device_mapping = list(object({
      deviceName = string
      ebs = object({
        encrypted           = bool
        volumeSize          = string
        volumeType          = string
        kmsKeyID            = optional(string)
        deleteOnTermination = bool
      })
    }))
  }))
 [] no
karpenter_nodepools List of Provisioner maps 
list(object({
    nodepool_name                     = string
    nodeclass_name                    = string
    karpenter_nodepool_node_labels    = map(string)
    karpenter_nodepool_annotations    = map(string)
    karpenter_nodepool_node_taints    = list(map(string))
    karpenter_nodepool_startup_taints = list(map(string))
    karpenter_requirements = list(object({
      key      = string
      operator = string
      values   = list(string)
      })
    )
    karpenter_nodepool_disruption = object({
      consolidation_policy = string
      consolidate_after    = string
      expire_after         = string
    })
    karpenter_nodepool_disruption_budgets = list(map(any))
    karpenter_nodepool_weight             = number
  }))
 
[
  {
    "karpenter_nodepool_annotations": {},
    "karpenter_nodepool_disruption": {
      "consolidate_after": "10m",
      "consolidation_policy": "WhenEmptyOrUnderutilized",
      "expire_after": "168h"
    },
    "karpenter_nodepool_disruption_budgets": [
      {
        "nodes": "10%"
      }
    ],
    "karpenter_nodepool_node_labels": {
      "bottlerocket.aws/updater-interface-version": "2.0.0"
    },
    "karpenter_nodepool_node_taints": [],
    "karpenter_nodepool_startup_taints": [],
    "karpenter_nodepool_weight": 10,
    "karpenter_requirements": [
      {
        "key": "karpenter.k8s.aws/instance-category",
        "operator": "In",
        "values": [
          "t",
          "m"
        ]
      },
      {
        "key": "karpenter.k8s.aws/instance-cpu",
        "operator": "In",
        "values": [
          "2",
          "4"
        ]
      },
      {
        "key": "karpenter.k8s.aws/instance-memory",
        "operator": "Gt",
        "values": [
          "2048"
        ]
      },
      {
        "key": "karpenter.k8s.aws/instance-generation",
        "operator": "Gt",
        "values": [
          "2"
        ]
      },
      {
        "key": "karpenter.sh/capacity-type",
        "operator": "In",
        "values": [
          "on-demand"
        ]
      },
      {
        "key": "kubernetes.io/arch",
        "operator": "In",
        "values": [
          "amd64"
        ]
      },
      {
        "key": "kubernetes.io/os",
        "operator": "In",
        "values": [
          "linux"
        ]
      }
    ],
    "nodeclass_name": "default",
    "nodepool_name": "default"
  }
]
 no
karpenter_pod_resources Karpenter Pod Resource 
object({
    requests = object({
      cpu    = string
      memory = string
    })
    limits = object({
      cpu    = string
      memory = string
    })
  })
 
{
  "limits": {
    "cpu": "1",
    "memory": "2Gi"
  },
  "requests": {
    "cpu": "1",
    "memory": "2Gi"
  }
}
 no
karpenter_upgrade Karpenter Upgrade bool false no
manage_aws_auth_configmap Determines whether to manage the contents of the aws-auth configmap. NOTE - make it true when authentication_mode = CONFIG_MAP bool false no
node_security_group_additional_rules List of additional security group rules to add to the node security group created. Set source_cluster_security_group = true inside rules to set the cluster_security_group as source any {} no
node_security_group_enable_recommended_rules Determines whether to enable recommended security group rules for the node security group created. This includes node-to-node TCP ingress on ephemeral ports and allows all egress traffic bool true no
only_critical_addons_enabled Enabling this option will taint default node group with CriticalAddonsOnly=true:NoSchedule taint. Changing this forces a new resource to be created. bool false no
role_mapping List of IAM roles to give access to the EKS cluster 
list(object({
    rolearn  = string
    username = string
    groups   = list(string)
  }))
 [] no
skip_asg_role Skip creating ASG Service Linked Role if it's already created bool false no
subnet_ids A list of subnet IDs where the EKS cluster (ENIs) will be provisioned along with the nodes/node groups. Node groups can be deployed within a different set of subnet IDs from within the node group configuration list(string) n/a yes
tags A map of tags to add to all resources map(string) {} no
user_mapping List of IAM Users to give access to the EKS Cluster 
list(object({
    userarn  = string
    username = string
    groups   = list(string)
  }))
 [] no
vpc_id VPC ID to deploy the cluster into string n/a yes
worker_security_group_name Worker security group name string null no
workers_iam_boundary IAM boundary for the workers IAM role, if any string null no
workers_iam_role Workers IAM Role name. If undefined, is the same as the cluster name suffixed with 'workers' string "" no

Outputs

Name Description
cluster_arn The ARN of the EKS cluster
cluster_certificate_authority_data Base64 Encoded Cluster CA Data
cluster_endpoint Endpoint of the EKS Cluster
cluster_iam_role_arn IAM Role ARN used by cluster
cluster_iam_role_name IAM Role Name used by Cluster
cluster_name EKS Cluster name created
cluster_oidc_issuer_url The URL on the EKS cluster for the OpenID Connect identity provider
cluster_platform_version Platform version of the EKS Cluster
cluster_primary_security_group_id Primary Security Group ID of the EKS cluster
cluster_security_group_id Security Group ID of the master nodes
cluster_version Version of the EKS Cluster
ebs_kms_key_arn KMS Key ARN used for EBS encryption
ebs_kms_key_id KMS Key ID used for EBS encryption
fargate_namespaces_for_security_group value for fargate_namespaces_for_security_group
oidc_provider_arn OIDC Provider ARN for IRSA
worker_iam_role_arn IAM Role ARN used by worker nodes
worker_iam_role_name IAM Role Name used by worker nodes
worker_security_group_id Security Group ID of the worker nodes