Skip to content

Latest commit

 

History

History
241 lines (184 loc) · 15.5 KB

README.md

File metadata and controls

241 lines (184 loc) · 15.5 KB
Raw

Google Cloud GKE On-Premises VMware Terraform Module

This Terraform module deploys Google Kubernetes Engine (GKE) On-Premises clusters on VMware vSphere infrastructure, with support for multiple node pools, load balancing, and RBAC configuration.

Features

  • Creates and manages GKE on-prem clusters on VMware vSphere
  • Configurable control plane with optional V2 features
  • Supports multiple node pools with autoscaling capabilities
  • Flexible IP management for worker nodes using IP ranges
  • MetalLB integration for load balancing with customizable address pools
  • RBAC configuration with admin and gateway user management
  • Anti-affinity group and auto-repair configuration
  • IP range validation and automatic IP allocation from ranges

Requirements

Name Version
terraform ~> 1.7
google ~> 6.12.0
kubernetes ~> 2.33.0

Usage

module "gke_onprem_vmware_cluster" {
  source = "telia-oss/gkeonprem/gcp"

  # Required parameters
  cluster_name             = "my-cluster"
  location                 = "us-west1"
  project_id               = "my-project"
  admin_cluster_membership = "projects/my-project/locations/us-west1/memberships/admin-cluster"
  admin_users              = ["[email protected]"]

  # vCenter configuration
  vcenter_config = {
    resource_pool = "/Datacenter/host/Cluster/Resources/Pool"
    folder        = "/Datacenter/vm/Folder"
  }

  # Network configuration
  network_config = {
    dns_servers           = ["8.8.8.8"]
    ntp_servers           = ["time.google.com"]
    vcenter_network       = "VM Network"
    control_plane_ips     = ["10.0.0.5", "10.0.0.6", "10.0.0.7"]
    worker_node_ip_ranges = ["10.0.0.20-10.0.0.30"]
    netmask               = "255.255.255.0"
    gateway               = "10.0.0.1"
  }

  # Load balancer configuration
  load_balancer_config = {
    control_plane_vip = "10.0.0.100"
    ingress_vip       = "10.0.0.101"
    address_pools = {
      "default-pool" = {
        addresses       = ["10.0.0.200-10.0.0.250"]
        manual_assign   = false
        avoid_buggy_ips = true
      }
    }
  }

  # Optional configurations
  control_plane_node = {
    cpus     = 4
    memory   = 8192
    replicas = 3
  }

  node_pools_config = {
    "default-pool" = {
      cpus              = 2
      memory_mb         = 4096
      replicas          = 3
      min_replicas      = 3
      max_replicas      = 4
      boot_disk_size_gb = 30
      image_type        = "cos_cgv2"
    }
    # Additional node pools can be defined here
  }

  connect_gateway_users   = ["[email protected]", "[email protected]"]
  gke_onprem_version      = "1.30.0-gke.1930"
  enable_control_plane_v2 = true
}

GKE On-Prem Cluster Configuration

Required Inputs

Name Description Type Default
cluster_name The name of the GKE on-prem cluster string n/a
location The GCP location where the cluster will be created string n/a
project_id The GCP project ID string n/a
admin_cluster_membership The admin cluster membership path string n/a
admin_users List of users for cluster admin access list(string) []
network_config Network configuration for the cluster
object
{
 service_address_cidr_blocks = optional(list(string), ["10.96.0.0/12"])
 pod_address_cidr_blocks = optional(list(string), ["192.168.0.0/16"])
 dns_servers = list(string)
 ntp_servers = list(string)
 vcenter_network = string
 control_plane_ips = list(string)
 worker_node_ip_ranges = list(string)
 netmask = string
 gateway = string
}
 n/a
load_balancer_config Load balancer configuration
object
{
 control_plane_vip = string
 ingress_vip = string
 address_pools = map(object({
 manual_assign = optional(bool, false)
 addresses = list(string)
 avoid_buggy_ips = optional(bool, true)
}))
}
 n/a
vcenter_config VMware vCenter configuration
object
{
 resource_pool = string
 folder = string
}
 n/a

Optional Inputs

Name Description Type Default
gke_onprem_version The version of GKE on-prem to install string "1.30.0-gke.1930"
enable_control_plane_v2 Whether to enable control plane v2 bool true
image_type The OS image type for nodes string "cos_cgv2"
admin_users List of users to grant cluster admin access list(string) []
connect_gateway_users List of users for GKE Connect Gateway access list(string) []
node_pools_config Map of node pool configurations
map(object)
{
  cpus              = optional(number, 2)
  memory_mb         = optional(number, 4096)
  replicas          = optional(number, 3)
  min_replicas      = optional(number, 3)
  max_replicas      = optional(number, 4)
  boot_disk_size_gb = optional(number, 30)
  image_type        = optional(string, "cos_cgv2")
}
 {}
control_plane_node Control plane node configuration
object
{
 cpus = optional(number, 4)
 memory = optional(number, 8192)
 replicas = optional(number, 3)
}
 
{
 cpus = 4
 memory = 8192
 replicas = 3
}

Notes

  • Collapsible Details: Inputs with complex types are wrapped in collapsible <details> blocks to reduce clutter while keeping the content easily accessible.
  • Ease of Maintenance: By using collapsible details, you can expand only the necessary sections for edits, reducing the likelihood of formatting errors.

Notes on Input Types

network_config

  • service_address_cidr_blocks:
    Optional list of CIDR blocks for service addresses.
    Default: ["10.96.0.0/12"].

  • pod_address_cidr_blocks:
    Optional list of CIDR blocks for pod addresses.
    Default: ["192.168.0.0/16"].

  • dns_servers:
    List of DNS server IP addresses (required).

  • ntp_servers:
    List of NTP server IP addresses (required).

  • vcenter_network:
    The VMware vCenter network name (required).

  • control_plane_ips:
    List of static IPs for control plane nodes (required).

  • worker_node_ip_ranges:
    List of IP ranges for worker nodes in the format "x.x.x.y-x.x.x.z" (required).

  • netmask:
    Netmask for the network (required).

  • gateway:
    Gateway IP address for the network (required).

vcenter_config

  • resource_pool:
    Full path to the vCenter resource pool (required).

  • folder:
    Full path to the vCenter folder (required).

load_balancer_config

  • control_plane_vip:
    Virtual IP for the control plane (required).

  • ingress_vip:
    Virtual IP for ingress (required).

  • address_pools:
    Map of address pool configurations (required). Each pool includes:

    • manual_assign:
      Optional boolean to manually assign IPs.
      Default: false.
    • addresses:
      List of IP addresses or ranges (required).
    • avoid_buggy_ips:
      Optional boolean to avoid buggy IPs.
      Default: true.

control_plane_node

  • cpus:
    Number of CPUs for control plane nodes.
    Optional, Default: 4.

  • memory:
    Memory in MB for control plane nodes.
    Optional, Default: 8192.

  • replicas:
    Number of control plane node replicas.
    Optional, Default: 3.

node_pools_config

Map where each key is a node pool name and the value is an object containing:

  • cpus:
    Number of CPUs.
    Optional, Default: 2.

  • memory_mb:
    Memory in MB.
    Optional, Default: 4096.

  • replicas:
    Number of replicas.
    Optional, Default: 1.

  • min_replicas:
    Minimum number of replicas for autoscaling.
    Optional, Default: 1.

  • max_replicas:
    Maximum number of replicas for autoscaling.
    Optional, Default: 3.

  • boot_disk_size_gb:
    Boot disk size in GB.
    Optional, Default: 30.

  • image_type:
    OS image type for nodes.
    Optional, Default: "cos_cgv2".
    Allowed values: "cos_cgv2", "cos", "ubuntu_cgv2", "ubuntu", "ubuntu_containerd", "windows".