PLAT-8294 : supporting private cluster (#56)

* adding Azure CNI Overlay support * updating vars description * updating documentation * adding support for private ACR and AKS * fixing precommit issues * changing dns zone name for aks and adding new role assignment * removing network rule * update dns zone format * Update Documentation * removing test resources * Docs update and module structure * removing dependency between acr and aks * fix vnet dependency issues * fixing tffmt issues * updating network vars defaults and descriptions * use existing subnet with public cluster when it is provided * Support private blob shared and flyte storage acounts (#62) * adding support for private storage * adding support for data endpoint when acr premium * fixing checks --------- Co-authored-by: Yassine MAACHI <[email protected]> * supporting private flyte resources * Move from apk to yum to test changes in latest ci image * adding condition to output --------- Co-authored-by: Yassine MAACHI <[email protected]> Co-authored-by: Dan Clegg <[email protected]>
dominodatalab · Sep 11, 2024 · c707eec · c707eec
1 parent 8fd47f4
commit c707eec
Show file tree

Hide file tree

Showing 18 changed files with 741 additions and 59 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -20,6 +20,31 @@ commands:
             echo export TF_VAR_deploy_id=\"${WORKSPACE}\" >> $BASH_ENV
             echo 'export TF_VAR_description="CircleCI Build for ${CIRCLE_PR_REPONAME}: ${CIRCLE_BUILD_URL}"' >> $BASH_ENV
             echo 'export WORKSPACE=azure-aks-circleci-${CIRCLE_BUILD_NUM}' >> $BASH_ENV
+  update_package_manager:
+    steps:
+      - run:
+          name: Update package manager
+          command: |
+            yum update
+            yum upgrade --assumeyes
+  install_git_azure_cli:
+    parameters:
+      terraform_version:
+        type: string
+        default: << pipeline.parameters.terraform_version >>
+    steps:
+      - run:
+          name: Install git
+          command: yum install git -y
+  install_tf_azure_cli:
+    parameters:
+      terraform_version:
+        type: string
+        default: << pipeline.parameters.terraform_version >>
+    steps:
+      - run:
+          name: Install terraform & jq
+          command: yum install terraform jq -y
   install_git:
     steps:
       - run:
@@ -100,7 +125,7 @@ jobs:
     environment: *default_environment
     steps:
       - checkout
-      - install_tf
+      - install_tf_azure_cli
       - set_env_vars
       - tf_init_apply
       - run:
@@ -123,8 +148,8 @@ jobs:
     environment: *default_environment
     steps:
       - checkout
-      - install_git
-      - install_tf
+      - install_git_azure_cli
+      - install_tf_azure_cli
       - set_env_vars
       - run:
           name: "Set module source to latest domino release"

diff --git a/README.md b/README.md
@@ -56,7 +56,11 @@ Please submit any feature enhancements, bug fixes, or ideas via pull requests or
 
 ## Modules
 
-No modules.
+| Name | Source | Version |
+|------|--------|---------|
+| <a name="module_domino_acr_ep"></a> [domino\_acr\_ep](#module\_domino\_acr\_ep) | ./modules/private_endpoint | n/a |
+| <a name="module_domino_blob_ep"></a> [domino\_blob\_ep](#module\_domino\_blob\_ep) | ./modules/private_endpoint | n/a |
+| <a name="module_domino_shared_ep"></a> [domino\_shared\_ep](#module\_domino\_shared\_ep) | ./modules/private_endpoint | n/a |
 
 ## Resources
 
@@ -70,23 +74,47 @@ No modules.
 | [azurerm_log_analytics_solution.logs](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/log_analytics_solution) | resource |
 | [azurerm_log_analytics_workspace.logs](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/log_analytics_workspace) | resource |
 | [azurerm_monitor_diagnostic_setting.control_plane](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/monitor_diagnostic_setting) | resource |
+| [azurerm_private_dns_zone.acr_private_dns_zone](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_dns_zone) | resource |
+| [azurerm_private_dns_zone.aks_private_dns_zone](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_dns_zone) | resource |
+| [azurerm_private_dns_zone.blob_private_dns_zone](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_dns_zone) | resource |
+| [azurerm_private_dns_zone.shared_private_dns_zone](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_dns_zone) | resource |
+| [azurerm_private_dns_zone_virtual_network_link.private_dns_zone_acr_vnet_link](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_dns_zone_virtual_network_link) | resource |
+| [azurerm_private_dns_zone_virtual_network_link.private_dns_zone_aks_vnet_link](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_dns_zone_virtual_network_link) | resource |
+| [azurerm_private_dns_zone_virtual_network_link.private_dns_zone_blob_vnet_link](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_dns_zone_virtual_network_link) | resource |
+| [azurerm_private_dns_zone_virtual_network_link.private_dns_zone_shared_vnet_link](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/private_dns_zone_virtual_network_link) | resource |
 | [azurerm_role_assignment.aks_domino_acr](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
+| [azurerm_role_assignment.aks_domino_private_acr](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
+| [azurerm_role_assignment.aks_domino_shared](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
+| [azurerm_role_assignment.aks_file_share_contributor](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
 | [azurerm_role_assignment.aks_network](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
 | [azurerm_role_assignment.hephaestus_acr](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
+| [azurerm_role_assignment.identity_assign_pdnsz](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
+| [azurerm_role_assignment.identity_assign_rg](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
+| [azurerm_role_assignment.identity_assign_vnet](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/role_assignment) | resource |
 | [azurerm_storage_account.domino](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account) | resource |
+| [azurerm_storage_account.domino_shared](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account) | resource |
+| [azurerm_storage_account_network_rules.domino_blob_rules](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account_network_rules) | resource |
+| [azurerm_storage_account_network_rules.domino_shared_rules](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_account_network_rules) | resource |
 | [azurerm_storage_container.domino_containers](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_container) | resource |
+| [azurerm_storage_share.shared_store](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_share) | resource |
+| [azurerm_user_assigned_identity.aks_assigned_identity](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/user_assigned_identity) | resource |
 | [azurerm_user_assigned_identity.hephaestus](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/user_assigned_identity) | resource |
 | [random_id.log_analytics_workspace_name_suffix](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/id) | resource |
 | [azurerm_kubernetes_service_versions.selected](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/kubernetes_service_versions) | data source |
 | [azurerm_resource_group.aks](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/resource_group) | data source |
+| [azurerm_subnet.aks_subnet](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/subnet) | data source |
 | [azurerm_subscription.current](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/subscription) | data source |
+| [azurerm_virtual_network.aks_vnet](https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/data-sources/virtual_network) | data source |
 
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_additional_node_pools"></a> [additional\_node\_pools](#input\_additional\_node\_pools) | additional node pools | <pre>map(object({<br>    enable_node_public_ip = optional(bool, false)<br>    vm_size               = string<br>    zones                 = list(string)<br>    node_labels           = map(string)<br>    node_os               = optional(string, "AzureLinux")<br>    node_taints           = optional(list(string), [])<br>    enable_auto_scaling   = optional(bool, true)<br>    min_count             = optional(number, 0)<br>    max_count             = number<br>    initial_count         = optional(number, 0)<br>    max_pods              = optional(number, 30)<br>    os_disk_size_gb       = optional(number, 128)<br>  }))</pre> | `{}` | no |
-| <a name="input_api_server_authorized_ip_ranges"></a> [api\_server\_authorized\_ip\_ranges](#input\_api\_server\_authorized\_ip\_ranges) | The IP ranges to whitelist for incoming traffic to the masters | `list(string)` | n/a | yes |
+| <a name="input_aks_subnet_name"></a> [aks\_subnet\_name](#input\_aks\_subnet\_name) | Subnet name for ACR/AKS, required when either private\_acr\_enabled or private\_cluster\_enabled is set to true. | `string` | `null` | no |
+| <a name="input_aks_vnet_name"></a> [aks\_vnet\_name](#input\_aks\_vnet\_name) | VNet name for ACR/AKS, required when either private\_acr\_enabled or private\_cluster\_enabled is set to true. | `string` | `null` | no |
+| <a name="input_aks_vnet_rg_name"></a> [aks\_vnet\_rg\_name](#input\_aks\_vnet\_rg\_name) | VNet Resource Groupe name for ACR/AKS, required when either private\_acr\_enabled or private\_cluster\_enabled is set to true. | `string` | `null` | no |
+| <a name="input_api_server_authorized_ip_ranges"></a> [api\_server\_authorized\_ip\_ranges](#input\_api\_server\_authorized\_ip\_ranges) | The IP ranges to whitelist for incoming traffic to the masters | `list(string)` | `null` | no |
 | <a name="input_cluster_sku_tier"></a> [cluster\_sku\_tier](#input\_cluster\_sku\_tier) | The Domino cluster SKU (defaults to Free) | `string` | `null` | no |
 | <a name="input_cni_overlay_enabled"></a> [cni\_overlay\_enabled](#input\_cni\_overlay\_enabled) | Flag to determine whether to use overlay network settings | `bool` | `false` | no |
 | <a name="input_containers"></a> [containers](#input\_containers) | storage containers to create | <pre>map(object({<br>    container_access_type = string<br>  }))</pre> | <pre>{<br>  "backups": {<br>    "container_access_type": "private"<br>  },<br>  "projects": {<br>    "container_access_type": "private"<br>  }<br>}</pre> | no |
@@ -99,6 +127,9 @@ No modules.
 | <a name="input_namespaces"></a> [namespaces](#input\_namespaces) | Namespace that are used for generating the service account bindings | `object({ platform = string, compute = string })` | n/a | yes |
 | <a name="input_node_pools"></a> [node\_pools](#input\_node\_pools) | default node pools | <pre>object({<br>    compute = object({<br>      enable_node_public_ip = optional(bool, false)<br>      vm_size               = optional(string, "Standard_D8s_v4")<br>      zones                 = optional(list(string), ["1", "2", "3"])<br>      node_labels = optional(map(string), {<br>        "dominodatalab.com/node-pool" = "default"<br>      })<br>      node_os             = optional(string, "AzureLinux")<br>      node_taints         = optional(list(string), [])<br>      enable_auto_scaling = optional(bool, true)<br>      min_count           = optional(number, 0)<br>      max_count           = optional(number, 10)<br>      initial_count       = optional(number, 1)<br>      max_pods            = optional(number, 30)<br>      os_disk_size_gb     = optional(number, 128)<br>    }),<br>    platform = object({<br>      enable_node_public_ip = optional(bool, false)<br>      vm_size               = optional(string, "Standard_D8s_v4")<br>      zones                 = optional(list(string), ["1", "2", "3"])<br>      node_labels = optional(map(string), {<br>        "dominodatalab.com/node-pool" = "platform"<br>      })<br>      node_os             = optional(string, "AzureLinux")<br>      node_taints         = optional(list(string), [])<br>      enable_auto_scaling = optional(bool, true)<br>      min_count           = optional(number, 1)<br>      max_count           = optional(number, 3)<br>      initial_count       = optional(number, 1)<br>      max_pods            = optional(number, 60)<br>      os_disk_size_gb     = optional(number, 128)<br>    }),<br>    gpu = object({<br>      enable_node_public_ip = optional(bool, false)<br>      vm_size               = optional(string, "Standard_NC6s_v3")<br>      zones                 = optional(list(string), [])<br>      node_labels = optional(map(string), {<br>        "dominodatalab.com/node-pool" = "default-gpu"<br>        "nvidia.com/gpu"              = "true"<br>      })<br>      node_os = optional(string, "AzureLinux")<br>      node_taints = optional(list(string), [<br>        "nvidia.com/gpu=true:NoExecute"<br>      ])<br>      enable_auto_scaling = optional(bool, true)<br>      min_count           = optional(number, 0)<br>      max_count           = optional(number, 1)<br>      initial_count       = optional(number, 0)<br>      max_pods            = optional(number, 30)<br>      os_disk_size_gb     = optional(number, 128)<br>    })<br>    system = object({<br>      enable_node_public_ip = optional(bool, false)<br>      vm_size               = optional(string, "Standard_DS4_v2")<br>      zones                 = optional(list(string), ["1", "2", "3"])<br>      node_labels           = optional(map(string), {})<br>      node_os               = optional(string, "AzureLinux")<br>      node_taints           = optional(list(string), [])<br>      enable_auto_scaling   = optional(bool, true)<br>      min_count             = optional(number, 1)<br>      max_count             = optional(number, 6)<br>      initial_count         = optional(number, 1)<br>      max_pods              = optional(number, 60)<br>      os_disk_size_gb       = optional(number, 128)<br>    })<br>  })</pre> | <pre>{<br>  "compute": {},<br>  "gpu": {},<br>  "platform": {},<br>  "system": {}<br>}</pre> | no |
 | <a name="input_pod_cidr"></a> [pod\_cidr](#input\_pod\_cidr) | CIDR block for Kubernetes pods, used when CNI Overlay is enabled | `string` | `"192.168.0.0/16"` | no |
+| <a name="input_private_acr_enabled"></a> [private\_acr\_enabled](#input\_private\_acr\_enabled) | Flag to determine whether to deploy a private ACR | `bool` | `false` | no |
+| <a name="input_private_cluster_enabled"></a> [private\_cluster\_enabled](#input\_private\_cluster\_enabled) | Flag to determine whether to deploy a private AKS | `bool` | `false` | no |
+| <a name="input_private_cluster_public_fqdn_enabled"></a> [private\_cluster\_public\_fqdn\_enabled](#input\_private\_cluster\_public\_fqdn\_enabled) | Flag to determine whether to use a public FQDN when deploying a private AKS cluster | `bool` | `null` | no |
 | <a name="input_registry_tier"></a> [registry\_tier](#input\_registry\_tier) | registry tier | `string` | `"Standard"` | no |
 | <a name="input_resource_group"></a> [resource\_group](#input\_resource\_group) | Name or id of optional pre-existing resource group to install AKS in | `string` | n/a | yes |
 | <a name="input_service_cidr"></a> [service\_cidr](#input\_service\_cidr) | CIDR block for Kubernetes services, used  when CNI Overlay is enabled | `string` | `"100.97.0.0/16"` | no |
@@ -111,9 +142,12 @@ No modules.
 | Name | Description |
 |------|-------------|
 | <a name="output_aks_identity"></a> [aks\_identity](#output\_aks\_identity) | AKS managed identity |
+| <a name="output_blob_dns_zone_name"></a> [blob\_dns\_zone\_name](#output\_blob\_dns\_zone\_name) | blob dns zone name |
 | <a name="output_containers"></a> [containers](#output\_containers) | storage details |
 | <a name="output_domino_acr"></a> [domino\_acr](#output\_domino\_acr) | Azure Container Registry details |
 | <a name="output_oidc_issuer_url"></a> [oidc\_issuer\_url](#output\_oidc\_issuer\_url) | OIDC issuer url |
+| <a name="output_private_cluster_enabled"></a> [private\_cluster\_enabled](#output\_private\_cluster\_enabled) | Flag to determine if AKS is private or public |
+| <a name="output_shared_storage_account"></a> [shared\_storage\_account](#output\_shared\_storage\_account) | shared storage account |
 | <a name="output_storage_account"></a> [storage\_account](#output\_storage\_account) | storage account |
 | <a name="output_workload_identities"></a> [workload\_identities](#output\_workload\_identities) | service identities |
 <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->