Skip to content

Commit

Permalink
feat (infra): [cluster-stamp] migrate arm json to bicep and upgrade k…
Browse files Browse the repository at this point in the history
…8s version (#65)

Co-authored-by: Chad Kittel <[email protected]>
  • Loading branch information
ferantivero and ckittel authored Oct 21, 2022
1 parent 1c71b40 commit 4680d4a
Show file tree
Hide file tree
Showing 8 changed files with 4,642 additions and 4,916 deletions.
2,238 changes: 2,238 additions & 0 deletions cluster-stamp.bicep

Large diffs are not rendered by default.

2,447 changes: 0 additions & 2,447 deletions cluster-stamp.json

This file was deleted.

2,249 changes: 2,249 additions & 0 deletions cluster-stamp.v2.bicep

Large diffs are not rendered by default.

2,458 changes: 0 additions & 2,458 deletions cluster-stamp.v2.json

This file was deleted.

10 changes: 5 additions & 5 deletions docs/deploy/09-aks-cluster.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,21 +45,21 @@ Now that the [hub-spoke network is provisioned](./08-cluster-networking.md), the
```bash
# [This takes about 20 minutes to run.]
az deployment group create -g rg-bu0001a0005 -f cluster-stamp.json -p targetVnetResourceId=${RESOURCEID_VNET_CLUSTERSPOKE} clusterAdminAadGroupObjectId=${AADOBJECTID_GROUP_CLUSTERADMIN} k8sControlPlaneAuthorizationTenantId=${TENANTID_K8SRBAC} appGatewayListenerCertificate=${APP_GATEWAY_LISTENER_CERTIFICATE_BASE64} aksIngressControllerCertificate=${INGRESS_CONTROLLER_CERTIFICATE_BASE64} jumpBoxImageResourceId=${RESOURCEID_IMAGE_JUMPBOX} jumpBoxCloudInitAsBase64=${CLOUDINIT_BASE64}
az deployment group create -g rg-bu0001a0005 -f cluster-stamp.bicep -p targetVnetResourceId=${RESOURCEID_VNET_CLUSTERSPOKE} clusterAdminAadGroupObjectId=${AADOBJECTID_GROUP_CLUSTERADMIN} k8sControlPlaneAuthorizationTenantId=${TENANTID_K8SRBAC} appGatewayListenerCertificate=${APP_GATEWAY_LISTENER_CERTIFICATE_BASE64} aksIngressControllerCertificate=${INGRESS_CONTROLLER_CERTIFICATE_BASE64} jumpBoxImageResourceId=${RESOURCEID_IMAGE_JUMPBOX} jumpBoxCloudInitAsBase64=${CLOUDINIT_BASE64}
# Or if you updated and wish to use the parameters file …
#az deployment group create -g rg-bu0001a0005 -f cluster-stamp.json -p "@azuredeploy.parameters.prod.json"
#az deployment group create -g rg-bu0001a0005 -f cluster-stamp.bicep -p "@azuredeploy.parameters.prod.json"
```
1. Update cluster deployment with managed identity assignments.
**cluster-stamp.v2.json** is a _tiny_ evolution of the **cluster-stamp.json** ARM template you literally just deployed in the step above. Because we are using Azure AD Pod Identity v1 as a Microsoft-managed add-on, the mechanism to associate identities with the cluster is via ARM template instead of via Kubernetes manifest deployments (as you would do with the vanilla open source solution). However, due to a current limitation of the add-on, managed identities for Pod Managed Identities CANNOT be associated to the cluster when the cluster is first being created, only as an update to an existing cluster. So this deployment will re-deploy with the Pod Managed Identity association as the _only change_. Pod Managed Identity v2 is in development and it will support assignment at cluster-creation time. This implementation will evolve to use Azure AD Pod Identity v2 when available and we'll remove this step and add the assignment directly in `cluster-stamp.json`.
**cluster-stamp.v2.json** is a _tiny_ evolution of the **cluster-stamp.bicep** ARM template you literally just deployed in the step above. Because we are using Azure AD Pod Identity v1 as a Microsoft-managed add-on, the mechanism to associate identities with the cluster is via ARM template instead of via Kubernetes manifest deployments (as you would do with the vanilla open source solution). However, due to a current limitation of the add-on, managed identities for Pod Managed Identities CANNOT be associated to the cluster when the cluster is first being created, only as an update to an existing cluster. So this deployment will re-deploy with the Pod Managed Identity association as the _only change_. Pod Managed Identity v2 is in development and it will support assignment at cluster-creation time. This implementation will evolve to use Azure AD Pod Identity v2 when available and we'll remove this step and add the assignment directly in `cluster-stamp.bicep`.

> :eyes: If you're curious to see what changed in the cluster stamp, [view the diff](https://diffviewer.azureedge.net/?l=https://raw.githubusercontent.com/mspnp/aks-baseline-regulated/main/cluster-stamp.json&r=https://raw.githubusercontent.com/mspnp/aks-baseline-regulated/main/cluster-stamp.v2.json).
> :eyes: If you're curious to see what changed in the cluster stamp, [view the diff](https://diffviewer.azureedge.net/?l=https://raw.githubusercontent.com/mspnp/aks-baseline-regulated/main/cluster-stamp.bicep&r=https://raw.githubusercontent.com/mspnp/aks-baseline-regulated/main/cluster-stamp.v2.bicep).
```bash
# [This takes about five minutes to run.]
az deployment group create -g rg-bu0001a0005 -f cluster-stamp.v2.json -p targetVnetResourceId=${RESOURCEID_VNET_CLUSTERSPOKE} clusterAdminAadGroupObjectId=${AADOBJECTID_GROUP_CLUSTERADMIN} k8sControlPlaneAuthorizationTenantId=${TENANTID_K8SRBAC} appGatewayListenerCertificate=${APP_GATEWAY_LISTENER_CERTIFICATE_BASE64} aksIngressControllerCertificate=${AKS_INGRESS_CONTROLLER_CERTIFICATE_BASE64} jumpBoxImageResourceId=${RESOURCEID_IMAGE_JUMPBOX} jumpBoxCloudInitAsBase64=${CLOUDINIT_BASE64}
az deployment group create -g rg-bu0001a0005 -f cluster-stamp.v2.bicep -p targetVnetResourceId=${RESOURCEID_VNET_CLUSTERSPOKE} clusterAdminAadGroupObjectId=${AADOBJECTID_GROUP_CLUSTERADMIN} k8sControlPlaneAuthorizationTenantId=${TENANTID_K8SRBAC} appGatewayListenerCertificate=${APP_GATEWAY_LISTENER_CERTIFICATE_BASE64} aksIngressControllerCertificate=${AKS_INGRESS_CONTROLLER_CERTIFICATE_BASE64} jumpBoxImageResourceId=${RESOURCEID_IMAGE_JUMPBOX} jumpBoxCloudInitAsBase64=${CLOUDINIT_BASE64}
# Or if you used the parameters file …
#az deployment group create -g rg-bu0001a0005 -f cluster-stamp.v2.json -p "@azuredeploy.parameters.prod.json"
Expand Down
144 changes: 144 additions & 0 deletions modules/ensureClusterIdentityHasRbacToSelfManagedResources.bicep
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
targetScope = 'resourceGroup'

/*** PARAMETERS ***/

@description('The AKS Control Plane Principal Id to be given with Network Contributor Role in different spoke subnets, so it can join VMSS and load balancers resources to them.')
@minLength(36)
@maxLength(36)
param miClusterControlPlanePrincipalId string

@description('The AKS Control Plane Principal Name to be used to create unique role assignments names.')
@minLength(3)
@maxLength(128)
param clusterControlPlaneIdentityName string

@description('The regional network spoke VNet Resource name that the cluster is being joined to, so it can be used to discover subnets during role assignments.')
@minLength(1)
param vnetSpokeName string

@allowed([
'australiaeast'
'canadacentral'
'centralus'
'eastus'
'eastus2'
'westus2'
'francecentral'
'germanywestcentral'
'northeurope'
'southafricanorth'
'southcentralus'
'uksouth'
'westeurope'
'japaneast'
'southeastasia'
])
@description('AKS Service, Node Pools, and supporting services (KeyVault, App Gateway, etc) region. This needs to be the same region as the vnet provided in these parameters.')
@minLength(4)
param location string

/*** EXISTING SUBSCRIPTION RESOURCES ***/

resource networkContributorRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: '4d97b98b-1d4f-4787-a291-c67834d212e7'
scope: subscription()
}

resource dnsZoneContributorRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
name: 'b12aa53e-6015-4669-85d0-8515ebb3ae7f'
scope: subscription()
}

/*** EXISTING SPOKE RESOURCES ***/

resource pdzMc 'Microsoft.Network/privateDnsZones@2020-06-01' existing = {
name: 'privatelink.${location}.azmk8s.io'
}

resource vnetSpoke 'Microsoft.Network/virtualNetworks@2022-01-01' existing = {
name: vnetSpokeName

resource snetClusterSystemNodePools 'subnets' existing = {
name: 'snet-cluster-systemnodepool'
}

resource snetClusterInScopeNodePools 'subnets' existing = {
name: 'snet-cluster-inscopenodepools'
}

resource snetClusterOutofScopeNodePools 'subnets' existing = {
name: 'snet-cluster-outofscopenodepools'
}

resource snetClusterIngressServices 'subnets' existing = {
name: 'snet-cluster-ingressservices'
}
}

/*** RESOURCES ***/

resource vnetMiClusterControlPlaneDnsZoneContributorRole_roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
scope: vnetSpoke
name: guid(vnetSpoke.id, dnsZoneContributorRole.id, clusterControlPlaneIdentityName)
properties: {
roleDefinitionId: dnsZoneContributorRole.id
description: 'Allows cluster identity to attach custom DNS zone with Private Link information to this virtual network.'
principalId: miClusterControlPlanePrincipalId
principalType: 'ServicePrincipal'
}
}

resource snetSystemNodePoolSubnetMiClusterControlPlaneNetworkContributorRole_roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
scope: vnetSpoke::snetClusterSystemNodePools
name: guid(vnetSpoke::snetClusterSystemNodePools.id, networkContributorRole.id, clusterControlPlaneIdentityName)
properties: {
roleDefinitionId: networkContributorRole.id
description: 'Allows cluster identity to join the nodepool vmss resources to this subnet.'
principalId: miClusterControlPlanePrincipalId
principalType: 'ServicePrincipal'
}
}

resource snetInScopeNodePoolSubnetsnetSystemNodePoolSubnetMiClusterControlPlaneNetworkContributorRole_roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
scope: vnetSpoke::snetClusterInScopeNodePools
name: guid(vnetSpoke::snetClusterInScopeNodePools.id, networkContributorRole.id, clusterControlPlaneIdentityName)
properties: {
roleDefinitionId: networkContributorRole.id
description: 'Allows cluster identity to join the nodepool vmss resources to this subnet.'
principalId: miClusterControlPlanePrincipalId
principalType: 'ServicePrincipal'
}
}

resource snetOutOfScopeNodePoolSubnetMiClusterControlPlaneNetworkContributorRole_roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
scope: vnetSpoke::snetClusterOutofScopeNodePools
name: guid(vnetSpoke::snetClusterOutofScopeNodePools.id, networkContributorRole.id, clusterControlPlaneIdentityName)
properties: {
roleDefinitionId: networkContributorRole.id
description: 'Allows cluster identity to join the nodepool vmss resources to this subnet.'
principalId: miClusterControlPlanePrincipalId
principalType: 'ServicePrincipal'
}
}

resource snetIngressServicesSubnetMiClusterControlPlaneNetworkContributorRole_roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
scope: vnetSpoke::snetClusterIngressServices
name: guid(vnetSpoke::snetClusterIngressServices.id, networkContributorRole.id, clusterControlPlaneIdentityName)
properties: {
roleDefinitionId: networkContributorRole.id
description: 'Allows cluster identity to join load balancers (ingress resources) to this subnet.'
principalId: miClusterControlPlanePrincipalId
principalType: 'ServicePrincipal'
}
}

resource pdzMcPrivatelinkAzmk8sIoMiClusterControlPlaneDnsZoneContributorRole_roleAssignment 'Microsoft.Authorization/roleAssignments@2020-04-01-preview' = {
scope: pdzMc
name: guid(pdzMc.id, dnsZoneContributorRole.id, clusterControlPlaneIdentityName)
properties: {
roleDefinitionId: dnsZoneContributorRole.id
description: 'Allows cluster identity to manage zone Entries for cluster\'s Private Link configuration.'
principalId: miClusterControlPlanePrincipalId
principalType: 'ServicePrincipal'
}
}
10 changes: 5 additions & 5 deletions networking/hub-region.v2.bicep
Original file line number Diff line number Diff line change
Expand Up @@ -892,7 +892,7 @@ resource hubFirewall 'Microsoft.Network/azureFirewalls@2021-05-01' = {
name: 'az-login'
description: 'Allow jumpboxes to perform az login.'
sourceIpGroups: [
aks_ipgroup.id
aksJumpbox_ipgroup.id
]
protocols: [
{
Expand All @@ -909,7 +909,7 @@ resource hubFirewall 'Microsoft.Network/azureFirewalls@2021-05-01' = {
name: 'az-management-api'
description: 'Allow jumpboxes to communicate with Azure management APIs.'
sourceIpGroups: [
aks_ipgroup.id
aksJumpbox_ipgroup.id
]
protocols: [
{
Expand All @@ -926,7 +926,7 @@ resource hubFirewall 'Microsoft.Network/azureFirewalls@2021-05-01' = {
name: 'az-cli-extensions'
description: 'Allow jumpboxes query az cli status and download extensions'
sourceIpGroups: [
aks_ipgroup.id
aksJumpbox_ipgroup.id
]
protocols: [
{
Expand All @@ -946,7 +946,7 @@ resource hubFirewall 'Microsoft.Network/azureFirewalls@2021-05-01' = {
name: 'github'
description: 'Allow pulling things down from GitHub. [Only a requirement of this walkthrough because we deploy some manifests that you clone from your repo.]'
sourceIpGroups: [
aks_ipgroup.id
aksJumpbox_ipgroup.id
]
protocols: [
{
Expand All @@ -964,7 +964,7 @@ resource hubFirewall 'Microsoft.Network/azureFirewalls@2021-05-01' = {
name: 'azure-monitor-addon'
description: 'Required for Azure Monitor Extension on Jumpbox.'
sourceIpGroups: [
aks_ipgroup.id
aksJumpbox_ipgroup.id
]
protocols: [
{
Expand Down
2 changes: 1 addition & 1 deletion networking/spoke-BU0001A0005-01.bicep
Original file line number Diff line number Diff line change
Expand Up @@ -862,7 +862,7 @@ resource aksPrivateDnsZones_virtualNetworkLink_toClusterVNet 'Microsoft.Network/
location: 'global'
properties: {
virtualNetwork: {
id: clusterVNet.id
id: hubVnetResourceId
}
registrationEnabled: false
}
Expand Down

0 comments on commit 4680d4a

Please sign in to comment.