Update to 2.22.4 (#55)

* Update to AKS 1.22 & Nginx 1.1.0

* Update AKV provider to non-alpha version

* Better handling of NetworkWatcher RG

README.md

@@ -33,7 +33,7 @@ Finally, this implementation uses a small, custom application as an example work
 
 #### Azure platform
 
-* AKS v1.21
+* AKS v1.22
   * System and User [node pool separation](https://docs.microsoft.com/azure/aks/use-system-pools)
   * [AKS-managed Azure AD](https://docs.microsoft.com/azure/aks/managed-aad)
   * Managed Identities for kubelet and control plane

cluster-manifests/ingress-nginx/akv-tls-provider.yaml

@@ -1,4 +1,4 @@
-apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
+apiVersion: secrets-store.csi.x-k8s.io/v1
 kind: SecretProviderClass
 metadata:
   name: aks-ingress-contoso-com-tls-secret-csi-akv

cluster-manifests/ingress-nginx/deployment.yaml

@@ -23,6 +23,7 @@ data:
   # the ingress controller directly is AAG.
   whitelist-source-range: 10.240.4.16/28
   ssl-redirect: "false"
+  allow-snippet-annotations: "true"
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRole
@@ -40,6 +41,7 @@ rules:
       - nodes
       - pods
       - secrets
+      - namespaces
     verbs:
       - list
       - watch
@@ -58,7 +60,6 @@ rules:
       - list
       - watch
   - apiGroups:
-      - extensions
       - networking.k8s.io
     resources:
       - ingresses
@@ -74,7 +75,6 @@ rules:
       - create
       - patch
   - apiGroups:
-      - extensions
       - networking.k8s.io
     resources:
       - ingresses/status
@@ -141,44 +141,42 @@ rules:
       - list
       - watch
   - apiGroups:
-      - extensions
       - networking.k8s.io
     resources:
       - ingresses
     verbs:
       - get
       - list
       - watch
-  - apiGroups:                                                                                                 
-      - extensions                                                                                                 
-      - networking.k8s.io                                                                                               
-    resources:                                                                                                 
-      - ingresses/status                                                                                                 
-    verbs:                                                                                                 
-      - update                                                                                                 
-  - apiGroups:                                                                                                 
-      - networking.k8s.io                                                                                               
-    resources:                                                                                                 
-      - ingressclasses                                                                                                 
-    verbs:                                                                                                 
-      - get                                                                                                 
-      - list                                                                                                 
-      - watch                                                                                                 
-  - apiGroups:                                                                                                 
-      - ''                                                                                                 
-    resources:                                                                                                 
-      - configmaps                                                                                                 
-    resourceNames:                                                                                                 
-      - ingress-controller-leader-nginx                                                                                                 
-    verbs:                                                                                                 
-      - get                                                                                                 
-      - update                                                                                                 
-  - apiGroups:                                                                                                 
-      - ''                                                                                                 
-    resources:                                                                                                 
-      - configmaps                                                                                                 
-    verbs:                                                                                                 
-      - create                                                                                                 
+  - apiGroups:
+      - networking.k8s.io
+    resources:
+      - ingresses/status
+    verbs:
+      - update
+  - apiGroups:
+      - networking.k8s.io
+    resources:
+      - ingressclasses
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - ''
+    resources:
+      - configmaps
+    resourceNames:
+      - ingress-controller-leader
+    verbs:
+      - get
+      - update
+  - apiGroups:
+      - ''
+    resources:
+      - configmaps
+    verbs:
+      - create
   - apiGroups:
       - ''
     resources:
@@ -220,6 +218,7 @@ spec:
     - name: https-webhook
       port: 443
       targetPort: webhook
+      appProtocol: https
   selector:
     app.kubernetes.io/name: ingress-nginx
     app.kubernetes.io/instance: ingress-nginx
@@ -241,11 +240,15 @@ spec:
   type: LoadBalancer
   loadBalancerIP: 10.240.4.4
   externalTrafficPolicy: Local
+  ipFamilyPolicy: SingleStack
+  ipFamilies: 
+    - IPv4
   ports:
     - name: https
       port: 443
       protocol: TCP
       targetPort: https
+      appProtocol: https
   selector:
     app.kubernetes.io/name: ingress-nginx
     app.kubernetes.io/instance: ingress-nginx
@@ -269,6 +272,7 @@ spec:
       app.kubernetes.io/instance: ingress-nginx
       app.kubernetes.io/component: controller
       pci-scope: out-of-scope
+  replicas: 1
   revisionHistoryLimit: 10
   minReadySeconds: 0
   template:
@@ -286,7 +290,7 @@ spec:
       dnsPolicy: ClusterFirst
       containers:
         - name: controller
-          image: YOUR_ACR.azurecr.io/live/ingress-nginx/controller:v0.49.0
+          image: YOUR_ACR.azurecr.io/live/ingress-nginx/controller:v1.1.0
           imagePullPolicy: IfNotPresent
           lifecycle:
             preStop:
@@ -297,12 +301,13 @@ spec:
             - /nginx-ingress-controller
             - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
             - --election-id=ingress-controller-leader
-            - --ingress-class=nginx
+            - --controller-class=k8s.io/ingress-nginx
             - --default-ssl-certificate=$(POD_NAMESPACE)/ingress-tls-csi
             - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
             - --validating-webhook=:8443
             - --validating-webhook-certificate=/usr/local/certificates/cert
             - --validating-webhook-key=/usr/local/certificates/key
+            - --v=3
           securityContext:
             capabilities:
               drop:
@@ -382,6 +387,17 @@ spec:
             volumeAttributes:
               secretProviderClass: aks-ingress-contoso-com-tls-secret-csi-akv
 ---
+apiVersion: networking.k8s.io/v1
+kind: IngressClass
+metadata:
+  labels:
+    app.kubernetes.io/name: ingress-nginx
+    app.kubernetes.io/instance: ingress-nginx
+    app.kubernetes.io/component: controller
+  name: nginx
+spec:
+  controller: k8s.io/ingress-nginx
+---
 apiVersion: admissionregistration.k8s.io/v1
 kind: ValidatingWebhookConfiguration
 metadata:
@@ -397,7 +413,7 @@ webhooks:
       - apiGroups:
           - networking.k8s.io
         apiVersions:
-          - v1beta1
+          - v1
         operations:
           - CREATE
           - UPDATE
@@ -407,12 +423,11 @@ webhooks:
     sideEffects: None
     admissionReviewVersions:
       - v1
-      - v1beta1
     clientConfig:
       service:
         namespace: ingress-nginx
         name: ingress-nginx-controller-admission
-        path: /networking/v1beta1/ingresses
+        path: /networking/v1/ingresses
 ---
 apiVersion: v1
 kind: ServiceAccount
@@ -535,6 +550,8 @@ spec:
             limits:
               cpu: 300m
               memory: 100Mi
+          securityContext:
+            allowPrivilegeEscalation: false
       nodeSelector:
         kubernetes.io/os: linux
         pci-scope: out-of-scope
@@ -587,6 +604,8 @@ spec:
             limits:
               cpu: 300m
               memory: 100Mi
+          securityContext:
+            allowPrivilegeEscalation: false
       nodeSelector:
         kubernetes.io/os: linux
         pci-scope: out-of-scope

cluster-stamp.json

@@ -103,7 +103,7 @@
         }
     },
     "variables": {
-        "kubernetesVersion": "1.21.2",
+        "kubernetesVersion": "1.22.4",
 
         "networkContributorRole": "[concat(subscription().Id, '/providers/Microsoft.Authorization/roleDefinitions/4d97b98b-1d4f-4787-a291-c67834d212e7')]",
         "monitoringMetricsPublisherRole": "[concat(subscription().Id, '/providers/Microsoft.Authorization/roleDefinitions/3913510d-42f4-4e42-8a64-420c390055eb')]",
@@ -302,7 +302,8 @@
                     "name": "DependencyAgentLinux",
                     "location": "[parameters('location')]",
                     "dependsOn": [
-                        "[resourceId('Microsoft.Compute/virtualMachineScaleSets', 'vmss-jumpboxes')]"
+                        "[resourceId('Microsoft.Compute/virtualMachineScaleSets', 'vmss-jumpboxes')]",
+                        "[resourceId('Microsoft.Compute/virtualMachineScaleSets/extensions', 'vmss-jumpboxes', 'OMSExtension')]"
                     ],
                     "properties": {
                         "publisher": "Microsoft.Azure.Monitoring.DependencyAgent",

cluster-stamp.v2.json

@@ -103,7 +103,7 @@
         }
     },
     "variables": {
-        "kubernetesVersion": "1.21.2",
+        "kubernetesVersion": "1.22.4",
 
         "networkContributorRole": "[concat(subscription().Id, '/providers/Microsoft.Authorization/roleDefinitions/4d97b98b-1d4f-4787-a291-c67834d212e7')]",
         "monitoringMetricsPublisherRole": "[concat(subscription().Id, '/providers/Microsoft.Authorization/roleDefinitions/3913510d-42f4-4e42-8a64-420c390055eb')]",
@@ -302,7 +302,8 @@
                     "name": "DependencyAgentLinux",
                     "location": "[parameters('location')]",
                     "dependsOn": [
-                        "[resourceId('Microsoft.Compute/virtualMachineScaleSets', 'vmss-jumpboxes')]"
+                        "[resourceId('Microsoft.Compute/virtualMachineScaleSets', 'vmss-jumpboxes')]",
+                        "[resourceId('Microsoft.Compute/virtualMachineScaleSets/extensions', 'vmss-jumpboxes', 'OMSExtension')]"
                     ],
                     "properties": {
                         "publisher": "Microsoft.Azure.Monitoring.DependencyAgent",

docs/deploy/04-subscription.md

@@ -85,7 +85,7 @@ Not only do we enable them in the steps below by default, but also set up an Azu
 1. Check for a pre-existing resource group with the name `networkWatcherRG`.
 
    ```bash
-   NETWORK_WATCHER_RG_REGION=$(az group list --query "[?name=='networkWatcherRG'].location" -o tsv)
+   NETWORK_WATCHER_RG_REGION=$(az group list --query "[?name=='networkWatcherRG' || name=='NetworkWatcherRG'].location" -o tsv)
    ```
 
    If your subscription is managed in such a way that Azure Network Watcher resources are found in a resource group other than the Azure default of `networkWatcherRG` or they do not use the Azure default `NetworkWatcher_<region>` naming convention, you will need to adjust the various ARM templates to compensate. Network Watchers are singletons (per region) in subscriptions, and organizations often manage them (and Flow Logs) via Azure Policy. This walkthrough assumes default naming conventions as set by Azure's [automatic deployment feature of Network Watchers](https://docs.microsoft.com/azure/network-watcher/network-watcher-create#network-watcher-is-automatically-enabled).

docs/deploy/10-pre-bootstrap.md

@@ -54,7 +54,7 @@ Using a security agent that is container-aware and can operate from within the c
    az acr import --source docker.io/falcosecurity/falco:0.29.1 -t quarantine/falcosecurity/falco:0.29.1 -n $ACR_NAME_QUARANTINE               && \
    az acr import --source docker.io/library/busybox:1.33.0 -t quarantine/library/busybox:1.33.0 -n $ACR_NAME_QUARANTINE                       && \
    az acr import --source docker.io/weaveworks/kured:1.9.0 -t quarantine/weaveworks/kured:1.9.0 -n $ACR_NAME_QUARANTINE                       && \
-   az acr import --source k8s.gcr.io/ingress-nginx/controller:v0.49.0 -t quarantine/ingress-nginx/controller:v0.49.0 -n $ACR_NAME_QUARANTINE  && \
+   az acr import --source k8s.gcr.io/ingress-nginx/controller:v1.1.0 -t quarantine/ingress-nginx/controller:v1.1.0 -n $ACR_NAME_QUARANTINE    && \
    az acr import --source docker.io/jettech/kube-webhook-certgen:v1.5.1 -t quarantine/jettech/kube-webhook-certgen:v1.5.1 -n $ACR_NAME_QUARANTINE
    ```
 
@@ -90,7 +90,7 @@ Using a security agent that is container-aware and can operate from within the c
    az acr import --source quarantine/falcosecurity/falco:0.29.1 -r $ACR_NAME_QUARANTINE -t live/falcosecurity/falco:0.29.1 -n $ACR_NAME                 && \
    az acr import --source quarantine/library/busybox:1.33.0 -r $ACR_NAME_QUARANTINE -t live/library/busybox:1.33.0 -n $ACR_NAME                         && \
    az acr import --source quarantine/weaveworks/kured:1.9.0 -r $ACR_NAME_QUARANTINE -t live/weaveworks/kured:1.9.0 -n $ACR_NAME                         && \
-   az acr import --source quarantine/ingress-nginx/controller:v0.49.0 -r $ACR_NAME_QUARANTINE -t live/ingress-nginx/controller:v0.49.0 -n $ACR_NAME     && \
+   az acr import --source quarantine/ingress-nginx/controller:v1.1.0 -r $ACR_NAME_QUARANTINE -t live/ingress-nginx/controller:v1.1.0 -n $ACR_NAME       && \
    az acr import --source quarantine/jettech/kube-webhook-certgen:v1.5.1 -r $ACR_NAME_QUARANTINE -t live/jettech/kube-webhook-certgen:v1.5.1 -n $ACR_NAME
    ```
 

docs/deploy/11-gitops.md

@@ -16,6 +16,10 @@ Your cluster was deployed with Azure Policy and Azure AD Pod Managed Identity. Y
 
 Your GitHub repo will be the source of truth for your cluster's configuration. Typically this would be a private repo, but for ease of demonstration, it'll be connected to a public repo (all firewall permissions are set to allow this specific interaction.) You'll be updating a configuration resource for Flux so that it knows to point to _your own repo_.
 
+## Alternative - Using the Flux AKS extension
+
+In the AKS Baseline, the cluster is [bootstrapped using the Flux AKS extension](https://github.com/mspnp/aks-baseline/blob/main/05-bootstrap-prep.md). That makes bootstrapping more "real time" with cluster deployment instead of as a post-deployment process that is described here. This reference implementation will be updated to use the Flux AKS extension at some point in time, which will further minimize the time period between cluster deployment and bootstrapping and removes the need to manually install Flux.
+
 ## Steps
 
 1. Update kustomization files to use images from your container registry.
@@ -111,13 +115,13 @@ Your GitHub repo will be the source of truth for your cluster's configuration. T
 
    ```output
    NAME                                  STATUS   ROLES   AGE   VERSION
-   aks-npinscope01-26621167-vmss000000   Ready    agent   20m   v1.21.x
-   aks-npinscope01-26621167-vmss000001   Ready    agent   20m   v1.21.x
-   aks-npooscope01-26621167-vmss000000   Ready    agent   20m   v1.21.x
-   aks-npooscope01-26621167-vmss000001   Ready    agent   20m   v1.21.x
-   aks-npsystem-26621167-vmss000000      Ready    agent   20m   v1.21.x
-   aks-npsystem-26621167-vmss000001      Ready    agent   20m   v1.21.x
-   aks-npsystem-26621167-vmss000002      Ready    agent   20m   v1.21.x
+   aks-npinscope01-26621167-vmss000000   Ready    agent   20m   v1.22.x
+   aks-npinscope01-26621167-vmss000001   Ready    agent   20m   v1.22.x
+   aks-npooscope01-26621167-vmss000000   Ready    agent   20m   v1.22.x
+   aks-npooscope01-26621167-vmss000001   Ready    agent   20m   v1.22.x
+   aks-npsystem-26621167-vmss000000      Ready    agent   20m   v1.22.x
+   aks-npsystem-26621167-vmss000001      Ready    agent   20m   v1.22.x
+   aks-npsystem-26621167-vmss000002      Ready    agent   20m   v1.22.x
    ```
 
    > :watch: The access tokens obtained in the prior two steps are subject to a Microsoft Identity Platform TTL (e.g. six hours). If your `az` or `kubectl` commands start erroring out after hours of usage with a message related to permission/authorization, you'll need to re-execute the `az login` and `az aks get-credentials` (overwriting your context) to refresh those tokens.