Feature/alzupgrade (#21)

* Support deployment of wayfinder in completely private environment --------- Co-authored-by: Kashif Saadat <[email protected]>
appvia · Nov 10, 2023 · 0d78cb5 · 0d78cb5
1 parent 63b04f9
commit 0d78cb5
Show file tree

Hide file tree

Showing 28 changed files with 573 additions and 45 deletions.
diff --git a/README.md b/README.md
@@ -67,23 +67,31 @@ The `terraform-docs` utility is used to generate this README. Follow the below s
 | <a name="input_aks_rbac_aad_admin_group_object_ids"></a> [aks\_rbac\_aad\_admin\_group\_object\_ids](#input\_aks\_rbac\_aad\_admin\_group\_object\_ids) | List of object IDs of the Azure AD groups that will be set as cluster admin. | `list(string)` | `[]` | no |
 | <a name="input_aks_sku_tier"></a> [aks\_sku\_tier](#input\_aks\_sku\_tier) | The SKU tier for this Kubernetes Cluster. | `string` | `"Standard"` | no |
 | <a name="input_aks_vnet_subnet_id"></a> [aks\_vnet\_subnet\_id](#input\_aks\_vnet\_subnet\_id) | The ID of the subnet in which to deploy the Kubernetes Cluster. | `string` | n/a | yes |
+| <a name="input_cert_manager_keyvault_cert_name"></a> [cert\_manager\_keyvault\_cert\_name](#input\_cert\_manager\_keyvault\_cert\_name) | Keyvault certificate name to use for cert-manager. Required if cluster issuer is keyvault | `string` | `null` | no |
+| <a name="input_cert_manager_keyvault_name"></a> [cert\_manager\_keyvault\_name](#input\_cert\_manager\_keyvault\_name) | Keyvault name to use for cert-manager. Required if cluster issuer is keyvault | `string` | `null` | no |
 | <a name="input_cluster_nodepool_version"></a> [cluster\_nodepool\_version](#input\_cluster\_nodepool\_version) | The Kubernetes version to use for the AKS cluster Nodepools. | `string` | `"1.26"` | no |
 | <a name="input_cluster_version"></a> [cluster\_version](#input\_cluster\_version) | The Kubernetes version to use for the AKS cluster. | `string` | `"1.26"` | no |
+| <a name="input_clusterissuer"></a> [clusterissuer](#input\_clusterissuer) | Cluster Issuer name to use for certs | `string` | `"letsencrypt-prod"` | no |
 | <a name="input_clusterissuer_email"></a> [clusterissuer\_email](#input\_clusterissuer\_email) | The email address to use for the cert-manager cluster issuer. | `string` | n/a | yes |
 | <a name="input_create_duration_delay"></a> [create\_duration\_delay](#input\_create\_duration\_delay) | Used to tune terraform apply when faced with errors caused by API caching or eventual consistency. Sets a custom delay period after creation of the specified resource type. | <pre>object({<br>    azurerm_role_definition         = optional(string, "180s")<br>    kubectl_manifest_cloud_identity = optional(string, "30s")<br>  })</pre> | `{}` | no |
 | <a name="input_create_localadmin_user"></a> [create\_localadmin\_user](#input\_create\_localadmin\_user) | Whether to create a localadmin user for access to the Wayfinder Portal and API | `bool` | `true` | no |
 | <a name="input_destroy_duration_delay"></a> [destroy\_duration\_delay](#input\_destroy\_duration\_delay) | Used to tune terraform destroy when faced with errors caused by API caching or eventual consistency. Sets a custom delay period after destruction of the specified resource type. | <pre>object({<br>    azurerm_role_definition         = optional(string, "0s")<br>    kubectl_manifest_cloud_identity = optional(string, "60s")<br>  })</pre> | `{}` | no |
 | <a name="input_disable_internet_access"></a> [disable\_internet\_access](#input\_disable\_internet\_access) | Whether to disable internet access for AKS and the Wayfinder ingress controller | `bool` | `false` | no |
 | <a name="input_disable_local_login"></a> [disable\_local\_login](#input\_disable\_local\_login) | Whether to disable local login for Wayfinder. Note: An IDP must be configured within Wayfinder, otherwise you will not be able to log in. | `bool` | `false` | no |
+| <a name="input_dns_provider"></a> [dns\_provider](#input\_dns\_provider) | DNS provider for External DNS | `string` | `"azure"` | no |
 | <a name="input_dns_resource_group_id"></a> [dns\_resource\_group\_id](#input\_dns\_resource\_group\_id) | The ID of the resource group where the DNS Zone exists, if different to Wayfinder's resource group. | `string` | `""` | no |
 | <a name="input_dns_zone_id"></a> [dns\_zone\_id](#input\_dns\_zone\_id) | The ID of the Azure DNS Zone to use. | `string` | n/a | yes |
 | <a name="input_dns_zone_name"></a> [dns\_zone\_name](#input\_dns\_zone\_name) | The name of the Azure DNS zone to use. | `string` | n/a | yes |
 | <a name="input_enable_k8s_resources"></a> [enable\_k8s\_resources](#input\_enable\_k8s\_resources) | Whether to enable the creation of Kubernetes resources for Wayfinder (helm and kubectl manifest deployments) | `bool` | `true` | no |
 | <a name="input_enable_wf_cloudaccess"></a> [enable\_wf\_cloudaccess](#input\_enable\_wf\_cloudaccess) | Whether to configure CloudIdentity and admin CloudAccessConfig resources in Wayfinder once installed (requires enable\_k8s\_resources) | `bool` | `true` | no |
 | <a name="input_environment"></a> [environment](#input\_environment) | The environment in which the resources are deployed. | `string` | `"production"` | no |
 | <a name="input_location"></a> [location](#input\_location) | The Azure region to use. | `string` | `"uksouth"` | no |
+| <a name="input_private_dns_zone_id"></a> [private\_dns\_zone\_id](#input\_private\_dns\_zone\_id) | Private DNS zone to use for private clusters | `string` | `null` | no |
 | <a name="input_resource_group_name"></a> [resource\_group\_name](#input\_resource\_group\_name) | The name of the resource group in which to create the AKS cluster. | `string` | n/a | yes |
 | <a name="input_tags"></a> [tags](#input\_tags) | A mapping of tags to assign to resources. | `map(string)` | `{}` | no |
+| <a name="input_user_assigned_identity"></a> [user\_assigned\_identity](#input\_user\_assigned\_identity) | MSI id for AKS to run as | `string` | n/a | yes |
+| <a name="input_venafi_apikey"></a> [venafi\_apikey](#input\_venafi\_apikey) | Venafi API key - required if using Venafi cluster issuer | `string` | `""` | no |
+| <a name="input_venafi_zone"></a> [venafi\_zone](#input\_venafi\_zone) | Venafi zone - required if using Venafi cluster issuer | `string` | `""` | no |
 | <a name="input_wayfinder_domain_name_api"></a> [wayfinder\_domain\_name\_api](#input\_wayfinder\_domain\_name\_api) | The domain name to use for the Wayfinder API (e.g. api.wayfinder.example.com) | `string` | n/a | yes |
 | <a name="input_wayfinder_domain_name_ui"></a> [wayfinder\_domain\_name\_ui](#input\_wayfinder\_domain\_name\_ui) | The domain name to use for the Wayfinder UI (e.g. portal.wayfinder.example.com) | `string` | n/a | yes |
 | <a name="input_wayfinder_idp_details"></a> [wayfinder\_idp\_details](#input\_wayfinder\_idp\_details) | The IDP details to use for Wayfinder to enable SSO | <pre>object({<br>    type          = string<br>    clientId      = optional(string)<br>    clientSecret  = optional(string)<br>    serverUrl     = optional(string)<br>    azureTenantId = optional(string)<br>  })</pre> | <pre>{<br>  "azureTenantId": "",<br>  "clientId": null,<br>  "clientSecret": null,<br>  "serverUrl": "",<br>  "type": "none"<br>}</pre> | no |
@@ -96,7 +104,7 @@ The `terraform-docs` utility is used to generate this README. Follow the below s
 
 | Name | Description |
 |------|-------------|
-| <a name="output_aks_admin_host"></a> [aks\_admin\_host](#output\_aks\_admin\_host) | The public API URL of the Azure Kubernetes Managed Cluster. |
+| <a name="output_aks_admin_host"></a> [aks\_admin\_host](#output\_aks\_admin\_host) | The API URL of the Azure Kubernetes Managed Cluster. |
 | <a name="output_aks_client_certificate"></a> [aks\_client\_certificate](#output\_aks\_client\_certificate) | The `client_certificate` in the `azurerm_kubernetes_cluster`'s `kube_admin_config` block.  Base64 encoded public certificate used by clients to authenticate to the Kubernetes cluster. |
 | <a name="output_aks_client_key"></a> [aks\_client\_key](#output\_aks\_client\_key) | The `client_key` in the `azurerm_kubernetes_cluster`'s `kube_admin_config` block. Base64 encoded private key used by clients to authenticate to the Kubernetes cluster. |
 | <a name="output_aks_cluster_ca_certificate"></a> [aks\_cluster\_ca\_certificate](#output\_aks\_cluster\_ca\_certificate) | The `cluster_ca_certificate` in the `azurerm_kubernetes_cluster`'s `kube_admin_config` block. Base64 encoded public CA certificate used as the root of trust for the Kubernetes cluster. |

diff --git a/aks.tf b/aks.tf
@@ -21,6 +21,7 @@ module "aks" {
   azure_policy_enabled                  = true
   enable_auto_scaling                   = true
   enable_host_encryption                = var.aks_enable_host_encryption
+  key_vault_secrets_provider_enabled    = var.clusterissuer == "keyvault" ? true : false
   kubernetes_version                    = var.cluster_version
   maintenance_window                    = var.aks_maintenance_window
   net_profile_dns_service_ip            = "192.168.100.10"
@@ -33,8 +34,11 @@ module "aks" {
   os_disk_size_gb                       = 50
   os_disk_type                          = "Ephemeral"
   os_sku                                = "Ubuntu"
+  net_profile_outbound_type             = var.disable_internet_access == true ? "userDefinedRouting" : "loadBalancer"
   private_cluster_enabled               = var.disable_internet_access
-  private_cluster_public_fqdn_enabled   = var.disable_internet_access
+  private_cluster_public_fqdn_enabled   = false
+  private_dns_zone_id                   = var.private_dns_zone_id
+  public_network_access_enabled         = !var.disable_internet_access
   rbac_aad                              = true
   rbac_aad_admin_group_object_ids       = var.aks_rbac_aad_admin_group_object_ids
   rbac_aad_managed                      = true
@@ -44,6 +48,8 @@ module "aks" {
   storage_profile_disk_driver_version   = "v1"
   tags                                  = local.tags
   vnet_subnet_id                        = var.aks_vnet_subnet_id
+  identity_ids                          = [var.user_assigned_identity]
+  identity_type                         = "UserAssigned"
   workload_identity_enabled             = true
 
   agents_pool_linux_os_configs = [

diff --git a/cert-manager.tf b/cert-manager.tf
@@ -6,6 +6,7 @@ resource "azurerm_user_assigned_identity" "cert_manager" {
 }
 
 resource "azurerm_role_assignment" "cert_manager_dns_contributor" {
+  count                = var.clusterissuer == "letsencrypt-prod" ? 1 : 0
   scope                = var.dns_zone_id
   role_definition_name = "DNS Zone Contributor"
   principal_id         = azurerm_user_assigned_identity.cert_manager.principal_id
@@ -17,6 +18,19 @@ resource "azurerm_role_assignment" "cert_manager_reader" {
   principal_id         = azurerm_user_assigned_identity.cert_manager.principal_id
 }
 
+data "azurerm_key_vault" "cert_kv" {
+  count               = var.clusterissuer == "keyvault" ? 1 : 0
+  name                = var.cert_manager_keyvault_name
+  resource_group_name = var.resource_group_name
+}
+
+resource "azurerm_role_assignment" "cert_manager_keyvault" {
+  count                = var.clusterissuer == "keyvault" ? 1 : 0
+  scope                = data.azurerm_key_vault.cert_kv[0].id
+  role_definition_name = "Key Vault Secrets User"
+  principal_id         = azurerm_user_assigned_identity.cert_manager.principal_id
+}
+
 resource "azurerm_federated_identity_credential" "cert_manager" {
   name                = azurerm_user_assigned_identity.cert_manager.name
   resource_group_name = azurerm_user_assigned_identity.cert_manager.resource_group_name
@@ -26,27 +40,43 @@ resource "azurerm_federated_identity_credential" "cert_manager" {
   subject             = "system:serviceaccount:cert-manager:cert-manager"
 }
 
+resource "kubectl_manifest" "certmanager_namespace" {
+  count = var.enable_k8s_resources ? 1 : 0
+
+  depends_on = [
+    module.aks,
+  ]
+
+  yaml_body = templatefile("${path.module}/manifests/namespace.yml.tpl", {
+    namespace = "cert-manager"
+  })
+}
+
 resource "helm_release" "cert_manager" {
   count = var.enable_k8s_resources ? 1 : 0
 
   depends_on = [
     module.aks,
+    kubectl_manifest.cert_manager_clusterissuer_keyvault_secret,
+    azurerm_role_assignment.cert_manager_keyvault
   ]
 
   namespace        = "cert-manager"
-  create_namespace = true
+  create_namespace = false
 
   name        = "cert-manager"
   repository  = "https://charts.jetstack.io"
   chart       = "cert-manager"
   version     = "v1.11.0"
   max_history = 5
 
-  values = [templatefile("${path.module}/manifests/cert-manager-values.yml.tpl", {})]
+  values = [templatefile("${path.module}/manifests/cert-manager-values.yml.tpl", {
+    clusterissuer = var.clusterissuer
+  }), var.clusterissuer == "keyvault" ? templatefile("${path.module}/manifests/cert-manager-csi-values.yml.tpl", {}) : ""]
 }
 
 resource "kubectl_manifest" "cert_manager_clusterissuer" {
-  count = var.enable_k8s_resources ? 1 : 0
+  count = var.enable_k8s_resources && var.clusterissuer == "letsencrypt-prod" ? 1 : 0
 
   depends_on = [
     module.aks,
@@ -61,3 +91,58 @@ resource "kubectl_manifest" "cert_manager_clusterissuer" {
     identity_client_id = azurerm_user_assigned_identity.cert_manager.client_id
   })
 }
+
+resource "kubectl_manifest" "cert_manager_clusterissuer_vaas" {
+  count = var.enable_k8s_resources && var.clusterissuer == "vaas-issuer" ? 1 : 0
+
+  depends_on = [
+    module.aks,
+    helm_release.cert_manager,
+    kubectl_manifest.cert_manager_clusterissuer_vaas_secret
+  ]
+
+  yaml_body = templatefile("${path.module}/manifests/cert-manager-clusterissuer-vaas.yml.tpl", {
+    venafi_zone = var.venafi_zone
+  })
+}
+
+resource "kubectl_manifest" "cert_manager_clusterissuer_vaas_secret" {
+  count = var.enable_k8s_resources && var.clusterissuer == "vaas-issuer" ? 1 : 0
+
+  depends_on = [
+    module.aks,
+    helm_release.cert_manager,
+  ]
+
+  yaml_body = templatefile("${path.module}/manifests/cert-manager-clusterissuer-vaas-secret.yml.tpl", {
+    venafi_apikey = var.venafi_apikey
+  })
+}
+
+resource "kubectl_manifest" "cert_manager_clusterissuer_keyvault" {
+  count = var.enable_k8s_resources && var.clusterissuer == "keyvault" ? 1 : 0
+
+  depends_on = [
+    module.aks,
+    helm_release.cert_manager,
+    kubectl_manifest.cert_manager_clusterissuer_keyvault_secret
+  ]
+
+  yaml_body = templatefile("${path.module}/manifests/cert-manager-clusterissuer-keyvault.yml.tpl", {})
+}
+
+resource "kubectl_manifest" "cert_manager_clusterissuer_keyvault_secret" {
+  count = var.enable_k8s_resources && var.clusterissuer == "keyvault" ? 1 : 0
+
+  depends_on = [
+    module.aks,
+    kubectl_manifest.certmanager_namespace
+  ]
+
+  yaml_body = templatefile("${path.module}/manifests/cert-manager-clusterissuer-keyvault-secret.yml.tpl", {
+    keyvault_name          = var.cert_manager_keyvault_name
+    keyvault_cert_name     = var.cert_manager_keyvault_cert_name
+    cert_manager_client_id = azurerm_user_assigned_identity.cert_manager.client_id
+    tenant_id              = data.azurerm_subscription.current.tenant_id
+  })
+}