This module handles opinionated Google Cloud Platform Kubernetes Engine cluster creation and configuration with Node Pools, IP MASQ, Network Policy, etc. The resources/services/activations/deletions that this module will create/trigger are:
- Create a GKE cluster with the provided addons
- Create GKE Node Pool(s) with provided configuration and attach to cluster
- Replace the default kube-dns configmap if
stub_domainsare provided - Activate network policy if
network_policyis true - Add
ip-masq-agentconfigmap with providednon_masquerade_cidrsifconfigure_ip_masqis true
Sub modules are provided from creating private clusters, beta private clusters, and beta public clusters as well. Beta sub modules allow for the use of various GKE beta features. See the modules directory for the various sub modules.
This module is meant for use with Terraform 0.12. If you haven't upgraded and need a Terraform 0.11.x-compatible version of this module, the last released version intended for Terraform 0.11.x is 3.0.0.
There are multiple examples included in the examples folder but simple usage is as follows:
module "gke" {
source = "terraform-google-modules/kubernetes-engine/google"
project_id = "<PROJECT ID>"
name = "gke-test-1"
region = "us-central1"
zones = ["us-central1-a", "us-central1-b", "us-central1-f"]
network = "vpc-01"
subnetwork = "us-central1-01"
ip_range_pods = "us-central1-01-gke-01-pods"
ip_range_services = "us-central1-01-gke-01-services"
http_load_balancing = false
horizontal_pod_autoscaling = true
kubernetes_dashboard = true
network_policy = true
node_pools = [
{
name = "default-node-pool"
machine_type = "n1-standard-2"
min_count = 1
max_count = 100
disk_size_gb = 100
disk_type = "pd-standard"
image_type = "COS"
auto_repair = true
auto_upgrade = true
service_account = "project-service-account@<PROJECT ID>.iam.gserviceaccount.com"
preemptible = false
initial_node_count = 80
},
]
node_pools_oauth_scopes = {
all = []
default-node-pool = [
"https://www.googleapis.com/auth/cloud-platform",
]
}
node_pools_labels = {
all = {}
default-node-pool = {
default-node-pool = true
}
}
node_pools_metadata = {
all = {}
default-node-pool = {
node-pool-metadata-custom-value = "my-node-pool"
}
}
node_pools_taints = {
all = []
default-node-pool = [
{
key = "default-node-pool"
value = true
effect = "PREFER_NO_SCHEDULE"
},
]
}
node_pools_tags = {
all = []
default-node-pool = [
"default-node-pool",
]
}
}Then perform the following commands on the root folder:
terraform initto get the pluginsterraform planto see the infrastructure planterraform applyto apply the infrastructure buildterraform destroyto destroy the built infrastructure
v3.0.0 is a breaking release. Refer to the Upgrading to v3.0 guide for details.
v2.0.0 is a breaking release. Refer to the Upgrading to v2.0 guide for details.
Version 1.0.0 of this module introduces a breaking change: adding the disable-legacy-endpoints metadata field to all node pools. This metadata is required by GKE and determines whether the /0.1/ and /v1beta1/ paths are available in the nodes' metadata server. If your applications do not require access to the node's metadata server, you can leave the default value of true provided by the module. If your applications require access to the metadata server, be sure to read the linked documentation to see if you need to set the value for this field to false to allow your applications access to the above metadata server paths.
In either case, upgrading to module version v1.0.0 will trigger a recreation of all node pools in the cluster.
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| basic_auth_password | The password to be used with Basic Authentication. | string | "" |
no |
| basic_auth_username | The username to be used with Basic Authentication. An empty value will disable Basic Authentication, which is the recommended configuration. | string | "" |
no |
| cluster_ipv4_cidr | The IP address range of the kubernetes pods in this cluster. Default is an automatically assigned CIDR. | string | "" |
no |
| cluster_resource_labels | The GCE resource labels (a map of key/value pairs) to be applied to the cluster | map(string) | <map> |
no |
| configure_ip_masq | Enables the installation of ip masquerading, which is usually no longer required when using aliasied IP addresses. IP masquerading uses a kubectl call, so when you have a private cluster, you will need access to the API server. | string | "false" |
no |
| create_service_account | Defines if service account specified to run nodes should be created. | bool | "true" |
no |
| description | The description of the cluster | string | "" |
no |
| disable_legacy_metadata_endpoints | Disable the /0.1/ and /v1beta1/ metadata server endpoints on the node. Changing this value will cause all node pools to be recreated. | bool | "true" |
no |
| grant_registry_access | Grants created cluster-specific service account storage.objectViewer role. | bool | "false" |
no |
| horizontal_pod_autoscaling | Enable horizontal pod autoscaling addon | bool | "true" |
no |
| http_load_balancing | Enable httpload balancer addon | bool | "true" |
no |
| initial_node_count | The number of nodes to create in this cluster's default node pool. | number | "0" |
no |
| ip_masq_link_local | Whether to masquerade traffic to the link-local prefix (169.254.0.0/16). | bool | "false" |
no |
| ip_masq_resync_interval | The interval at which the agent attempts to sync its ConfigMap file from the disk. | string | "60s" |
no |
| ip_range_pods | The name of the secondary subnet ip range to use for pods | string | n/a | yes |
| ip_range_services | The name of the secondary subnet range to use for services | string | n/a | yes |
| issue_client_certificate | Issues a client certificate to authenticate to the cluster endpoint. To maximize the security of your cluster, leave this option disabled. Client certificates don't automatically rotate and aren't easily revocable. WARNING: changing this after cluster creation is destructive! | bool | "false" |
no |
| kubernetes_dashboard | Enable kubernetes dashboard addon | bool | "false" |
no |
| kubernetes_version | The Kubernetes version of the masters. If set to 'latest' it will pull latest available version in the selected region. | string | "latest" |
no |
| logging_service | The logging service that the cluster should write logs to. Available options include logging.googleapis.com, logging.googleapis.com/kubernetes (beta), and none | string | "logging.googleapis.com" |
no |
| maintenance_start_time | Time window specified for daily maintenance operations in RFC3339 format | string | "05:00" |
no |
| master_authorized_networks_config | The desired configuration options for master authorized networks. The object format is {cidr_blocks = list(object({cidr_block = string, display_name = string}))}. Omit the nested cidr_blocks attribute to disallow external access (except the cluster node IPs, which GKE automatically whitelists). | object | <list> |
no |
| monitoring_service | The monitoring service that the cluster should write metrics to. Automatically send metrics from pods in the cluster to the Google Cloud Monitoring API. VM metrics will be collected by Google Compute Engine regardless of this setting Available options include monitoring.googleapis.com, monitoring.googleapis.com/kubernetes (beta) and none | string | "monitoring.googleapis.com" |
no |
| name | The name of the cluster (required) | string | n/a | yes |
| network | The VPC network to host the cluster in (required) | string | n/a | yes |
| network_policy | Enable network policy addon | bool | "false" |
no |
| network_policy_provider | The network policy provider. | string | "CALICO" |
no |
| network_project_id | The project ID of the shared VPC's host (for shared vpc support) | string | "" |
no |
| node_pools | List of maps containing node pools | list(map(string)) | <list> |
no |
| node_pools_labels | Map of maps containing node labels by node-pool name | map(map(string)) | <map> |
no |
| node_pools_metadata | Map of maps containing node metadata by node-pool name | map(map(string)) | <map> |
no |
| node_pools_oauth_scopes | Map of lists containing node oauth scopes by node-pool name | map(list(string)) | <map> |
no |
| node_pools_tags | Map of lists containing node network tags by node-pool name | map(list(string)) | <map> |
no |
| node_pools_taints | Map of lists containing node taints by node-pool name | object | <map> |
no |
| node_version | The Kubernetes version of the node pools. Defaults kubernetes_version (master) variable and can be overridden for individual node pools by setting the version key on them. Must be empyty or set the same as master at cluster creation. |
string | "" |
no |
| non_masquerade_cidrs | List of strings in CIDR notation that specify the IP address ranges that do not use IP masquerading. | list(string) | <list> |
no |
| project_id | The project ID to host the cluster in (required) | string | n/a | yes |
| region | The region to host the cluster in (required) | string | n/a | yes |
| regional | Whether is a regional cluster (zonal cluster if set false. WARNING: changing this after cluster creation is destructive!) | bool | "true" |
no |
| remove_default_node_pool | Remove default node pool while setting up the cluster | bool | "false" |
no |
| service_account | The service account to run nodes as if not overridden in node_pools. The create_service_account variable default value (true) will cause a cluster-specific service account to be created. |
string | "" |
no |
| stub_domains | Map of stub domains and their resolvers to forward DNS queries for a certain domain to an external DNS server | map(list(string)) | <map> |
no |
| subnetwork | The subnetwork to host the cluster in (required) | string | n/a | yes |
| upstream_nameservers | If specified, the values replace the nameservers taken by default from the node’s /etc/resolv.conf | list | <list> |
no |
| zones | The zones to host the cluster in (optional if regional cluster / required if zonal) | list(string) | <list> |
no |
| Name | Description |
|---|---|
| ca_certificate | Cluster ca certificate (base64 encoded) |
| endpoint | Cluster endpoint |
| horizontal_pod_autoscaling_enabled | Whether horizontal pod autoscaling enabled |
| http_load_balancing_enabled | Whether http load balancing enabled |
| kubernetes_dashboard_enabled | Whether kubernetes dashboard enabled |
| location | Cluster location (region if regional cluster, zone if zonal cluster) |
| logging_service | Logging service used |
| master_authorized_networks_config | Networks from which access to master is permitted |
| master_version | Current master kubernetes version |
| min_master_version | Minimum master kubernetes version |
| monitoring_service | Monitoring service used |
| name | Cluster name |
| network_policy_enabled | Whether network policy enabled |
| node_pools_names | List of node pools names |
| node_pools_versions | List of node pools versions |
| region | Cluster region |
| service_account | The service account to default running nodes as if not overridden in node_pools. |
| type | Cluster type (regional / zonal) |
| zones | List of zones in which the cluster resides |
Before this module can be used on a project, you must ensure that the following pre-requisites are fulfilled:
- Terraform and kubectl are installed on the machine where Terraform is executed.
- The Service Account you execute the module with has the right permissions.
- The Compute Engine and Kubernetes Engine APIs are active on the project you will launch the cluster in.
- If you are using a Shared VPC, the APIs must also be activated on the Shared VPC host project and your service account needs the proper permissions there.
The project factory can be used to provision projects with the correct APIs active and the necessary Shared VPC connections.
- kubectl 1.9.x
- Terraform 0.12
- Terraform Provider for GCP v2.9
In order to execute this module you must have a Service Account with the following project roles:
- roles/compute.viewer
- roles/container.clusterAdmin
- roles/container.developer
- roles/iam.serviceAccountAdmin
- roles/iam.serviceAccountUser
- roles/resourcemanager.projectIamAdmin (only required if
service_accountis set tocreate)
In order to operate with the Service Account you must activate the following APIs on the project where the Service Account was created:
- Compute Engine API - compute.googleapis.com
- Kubernetes Engine API - container.googleapis.com
The project has the following folders and files:
- /: root folder
- /examples: Examples for using this module and sub module.
- /helpers: Helper scripts.
- /scripts: Scripts for specific tasks on module (see Infrastructure section on this file).
- /test: Folders with files for testing the module (see Testing section on this file).
- /main.tf:
mainfile for the public module, contains all the resources to create. - /variables.tf: Variables for the public cluster module.
- /output.tf: The outputs for the public cluster module.
- /README.MD: This file.
- /modules: Private and beta sub modules.
To more cleanly handle cases where desired functionality would require complex duplication of Terraform resources (i.e. PR 51), this repository is largely generated from the autogen directory.
The root module is generated by running make generate. Changes to this repository should be made in the autogen directory where appropriate.
Note: The correct sequence to update the repo using autogen functionality is to run
make generate && make generate_docs. This will create the various Terraform files, and then
generate the Terraform documentation using terraform-docs.
- bundler
- gcloud
- terraform-docs 0.6.0
Run
make generate_docs
Integration tests are run though test-kitchen, kitchen-terraform, and InSpec.
Six test-kitchen instances are defined:
deploy-servicenode-poolshared-vpcsimple-regionalsimple-zonalstub-domains
The test-kitchen instances in test/fixtures/ wrap identically-named examples in the examples/ directory.
- Configure the test fixtures
- Download a Service Account key with the necessary permissions and put it in the module's root directory with the name
credentials.json.- Requires the permissions to run the module
- Requires
roles/compute.networkAdminto create the test suite's networks - Requires
roles/resourcemanager.projectIamAdminsince service account creation is tested
- Build the Docker container for testing:
make docker_build_kitchen_terraform
- Run the testing container in interactive mode:
make docker_run
The module root directory will be loaded into the Docker container at /cft/workdir/.
5. Run kitchen-terraform to test the infrastructure:
kitchen createcreates Terraform state and downloads modules, if applicable.kitchen convergecreates the underlying resources. Runkitchen converge <INSTANCE_NAME>to create resources for a specific test case.- Run
kitchen convergeagain. This is necessary due to an oddity in hownetworkPolicyConfigis handled by the upstream API. (See #72 for details). kitchen verifytests the created infrastructure. Runkitchen verify <INSTANCE_NAME>to run a specific test case.kitchen destroytears down the underlying resources created bykitchen converge. Runkitchen destroy <INSTANCE_NAME>to tear down resources for a specific test case.
Alternatively, you can simply run make test_integration_docker to run all the test steps non-interactively.
If you wish to parallelize running the test suites, it is also possible to offload the work onto Concourse to run each test suite for you using the command make test_integration_concourse. The .concourse directory will be created and contain all of the logs from the running test suites.
When running tests locally, you will need to use your own test project environment. You can configure your environment by setting all of the following variables:
export COMPUTE_ENGINE_SERVICE_ACCOUNT="<EXISTING_SERVICE_ACCOUNT>"
export PROJECT_ID="<PROJECT_TO_USE>"
export REGION="<REGION_TO_USE>"
export ZONES='["<LIST_OF_ZONES_TO_USE>"]'
export SERVICE_ACCOUNT_JSON="$(cat "<PATH_TO_SERVICE_ACCOUNT_JSON>")"
export CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE="<PATH_TO_SERVICE_ACCOUNT_JSON>"
export GOOGLE_APPLICATION_CREDENTIALS="<PATH_TO_SERVICE_ACCOUNT_JSON>"
Each test-kitchen instance is configured with a variables.tfvars file in the test fixture directory, e.g. test/fixtures/node_pool/terraform.tfvars.
For convenience, since all of the variables are project-specific, these files have been symlinked to test/fixtures/shared/terraform.tfvars.
Similarly, each test fixture has a variables.tf to define these variables, and an outputs.tf to facilitate providing necessary information for inspec to locate and query against created resources.
Each test-kitchen instance creates a GCP Network and Subnetwork fixture to house resources, and may create any other necessary fixture data as needed.
Run
make generate_docs
The makefile in this project will lint or sometimes just format any shell, Python, golang, Terraform, or Dockerfiles. The linters will only be run if the makefile finds files with the appropriate file extension.
All of the linter checks are in the default make target, so you just have to run
make -s
The -s is for 'silent'. Successful output looks like this
Running shellcheck
Running flake8
Running go fmt and go vet
Running terraform validate
Running hadolint on Dockerfiles
Checking for required files
Testing the validity of the header check
..
----------------------------------------------------------------------
Ran 2 tests in 0.026s
OK
Checking file headers
The following lines have trailing whitespace
The linters are as follows:
- Shell - shellcheck. Can be found in homebrew
- Python - flake8. Can be installed with 'pip install flake8'
- Golang - gofmt. gofmt comes with the standard golang installation. golang is a compiled language so there is no standard linter.
- Terraform - terraform has a built-in linter in the 'terraform validate' command.
- Dockerfiles - hadolint. Can be found in homebrew