diff --git a/docs/_posts/2024-11-05-afd-aks-ingress-tls.md b/docs/_posts/2024-11-05-afd-aks-ingress-tls.md new file mode 100644 index 0000000..8dab4a9 --- /dev/null +++ b/docs/_posts/2024-11-05-afd-aks-ingress-tls.md @@ -0,0 +1,593 @@ +--- +title: End to End TLS Encryption with AKS and AFD +description: Using Azure Front Door infront of an in cluster nginx ingress controller to provide end to end TLS encryption of application traffic. +authors: + - steve_griffith +--- + +# End to End TLS Encryption with AKS and AFD + +## Introduction + +In this walkthrough we'll create deploy an app with end to end TLS encryption, using Azure Front Door as the Internet Facing TLS endpoint and an Nginx Ingress controller running inside an AKS cluster as the backend. + +We'll use Azure Key Vault to store the TLS certificate, and will use the Key Vault CSI Driver to get the secrets into the ingress controller. The Key Vault CSI Driver will use Azure Workload Identity to safely retrieve the certificate. + +Let's get to it.... + +## Network Setup + +First, we'll need to establish the network where our AKS cluster will be deployed. Nothing special in our network design, other than the fact that I'm creating an Azure Network Security Group at the subnet level for added security. + +```bash +# Resource Group Creation +RG=E2ETLSLab +LOC=eastus2 + +# Create the Resource Group +az group create -g $RG -l $LOC + +# Get the resource group id +RG_ID=$(az group show -g $RG -o tsv --query id) + +# Set an environment variable for the VNet name +VNET_NAME=lablab-vnet +VNET_ADDRESS_SPACE=10.140.0.0/16 +AKS_SUBNET_ADDRESS_SPACE=10.140.0.0/24 + +# Create an NSG at the subnet level for security reasons +az network nsg create \ +--resource-group $RG \ +--name aks-subnet-nsg + +# Get the NSG ID +NSG_ID=$(az network nsg show -g $RG -n aks-subnet-nsg -o tsv --query id) + +# Create the Vnet along with the initial subet for AKS +az network vnet create \ +-g $RG \ +-n $VNET_NAME \ +--address-prefix $VNET_ADDRESS_SPACE \ +--subnet-name aks \ +--subnet-prefix $AKS_SUBNET_ADDRESS_SPACE \ +--network-security-group aks-subnet-nsg + +# Get a subnet resource ID, which we'll need when we create the AKS cluster +VNET_SUBNET_ID=$(az network vnet subnet show -g $RG --vnet-name $VNET_NAME -n aks -o tsv --query id) +``` + +## Cluster Creation + +Now, lets create the AKS cluster where our workload and ingress controller will reside. This will be a very plain AKS cluster, however we will deploy to our above created subnet and will enable the following features: + +- Workload Identity: This will be used by the Key Vault CSI Driver to retrieve the cluster certificate +- OIDC Issuer: This is required by Workload Identity to be used during service account fedration +- Key Vault CSI Driver: This will be used to retrieve the cluster certificate + +> *Note:* Since I created a NSG at the subnet level, I'll need to create a custom role which will be used later for automated private link creation. If you don't have an NSG on the subnet, and just rely on the managed NSG that AKS owns, then you don't need to create the custom role documented below. + +```bash +# NOTE: Make sure you give your cluster a unique name +CLUSTER_NAME=e2etlslab + +# Cluster Creation Command +az aks create \ +-g $RG \ +-n $CLUSTER_NAME \ +--node-count 2 \ +--enable-oidc-issuer \ +--enable-workload-identity \ +--enable-addons azure-keyvault-secrets-provider \ +--vnet-subnet-id $VNET_SUBNET_ID + +# Get the cluster identity +CLUSTER_IDENTITY=$(az aks show -g $RG -n $CLUSTER_NAME -o tsv --query identity.principalId) + +################################################################################################### +# Grant the cluster identity rights on the cluster nsg, which we'll need later when we create the +# private link. + +# NOTE: These steps are only needed if you have a custom NSG on the cluster subnet. + +# Create the role definition file +cat << EOF > pl-nsg-role.json +{ + "Name": "Private Link AKS Role", + "Description": "Grants the cluster rights on the NSG for Private Link Creation", + "Actions": [ + "Microsoft.Network/networkSecurityGroups/join/action" + ], + "NotActions": [], + "DataActions": [], + "NotDataActions": [], + "assignableScopes": [ + "${RG_ID}" + ] +} +EOF + +# Create the role definition in Azure +az role definition create --role-definition @pl-nsg-role.json + +# Assign the role +# NOTE: New role propegation may take a minute or to, so retry as needed +az role assignment create \ +--role "Private Link AKS Role" \ +--assignee $CLUSTER_IDENTITY \ +--scope $NSG_ID +################################################################################################### + + +# Get credentials +az aks get-credentials -g $RG -n $CLUSTER_NAME +``` + +## Setup Workload Identity + +Now that we have our cluster, lets finish setting up the workload identity that will be used to retrieve our certificate from Azure Key Vault. + +>*Note:* For simplicity, I'm keeping all resources in the 'default' namespace. You may want to modify this for your own deployment. + +```bash +# Set the namespace where we will deploy our app and ingress controller +NAMESPACE=default + +# Get the OIDC Issuer URL +export AKS_OIDC_ISSUER="$(az aks show -n $CLUSTER_NAME -g $RG --query "oidcIssuerProfile.issuerUrl" -otsv)" + +# Get the Tenant ID for later +export IDENTITY_TENANT=$(az account show -o tsv --query tenantId) + +# Create the managed identity +az identity create --name nginx-ingress-identity --resource-group $RG --location $LOC + +# Get identity client ID +export USER_ASSIGNED_CLIENT_ID=$(az identity show --resource-group $RG --name nginx-ingress-identity --query 'clientId' -o tsv) + +# Create a service account to federate with the managed identity +cat <*Note:* The Key Vault CSI driver uses the Kubernetes Container Storage Interface to initiate it's connection to Key Vault. That means, even though we really only care about having the certificate as a secret for use in our ingress definition, we still need to mount the secret as a volume to create the Kubernetes Secret. We'll mount the certificate secret to the ingress controller, but you could mount it to your app alternatively. Especially if you plan to use the certificate in your app directly as well. + +When you create a certificate in Azure Key Vault and then use the Key Vault CSI driver to access that certificate, you use the 'secret' object type and the certificate name. The returned secret contains the certificate key and crt file using the names tls.key and tls.crt. + +```bash +# We'll cat the SecretProviderClass direclty into kubectl +# You could also just cat it out to a file and use that file to deploy +cat << EOF | kubectl apply -f - +apiVersion: secrets-store.csi.x-k8s.io/v1 +kind: SecretProviderClass +metadata: + name: crashoverride-tls + namespace: ${NAMESPACE} +spec: + provider: azure + secretObjects: # secretObjects defines the desired state of synced K8s secret objects + - secretName: crashoverride-tls-csi + type: kubernetes.io/tls + data: + - objectName: crashoverride + key: crashoverride.key + - objectName: crashoverride + key: crashoverride.crt + parameters: + usePodIdentity: "false" + clientID: ${USER_ASSIGNED_CLIENT_ID} + keyvaultName: ${KEY_VAULT_NAME} # the name of the AKV instance + objects: | + array: + - | + objectName: crashoverride + objectType: secret + tenantId: ${IDENTITY_TENANT} # the tenant ID of the AKV instance +EOF +``` + +## Deploy the Ingress Controller + +Now we're ready to create our ingress controller. For our purposes, and for it's simplicity, we're going to use ingress-nginx. The approach will be roughly the same for any ingress controller. + +There are a few key points to note in the deployment below. + +1. We want our ingress controller to be on an internal network with no public IP, since Azure Front Door will provide the public endpoint. To do that we'll need to apply the 'azure-load-balancer-internal' annotation. +2. Azure Front Door can only connect to a private IP address using an Azure Private Link Service. Fortunately, AKS provides the 'azure-pls-create' annotation which will automatically create and manage a private link for you. +3. As mentioned above, since we're using the Key Vault CSI Driver, we need to mount the Secret Provider Class using the secret-store driver. + +```bash +helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx +helm repo update + +# Generate the values file we'll use to deploy ingress-nginx +cat < nginx-ingress-values.yaml +serviceAccount: + create: false + name: nginx-ingress-sa +controller: + replicaCount: 2 + service: + annotations: + service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path: /healthz + service.beta.kubernetes.io/azure-load-balancer-internal: "true" + service.beta.kubernetes.io/azure-pls-create: "true" + extraVolumes: + - name: crashoverride-secret-store + csi: + driver: secrets-store.csi.k8s.io + readOnly: true + volumeAttributes: + secretProviderClass: "crashoverride-tls" + extraVolumeMounts: + - name: crashoverride-secret-store + mountPath: "/mnt/crashoverride" + readOnly: true +EOF + +# Deploy ingress-nginx +helm install e2elab-ic ingress-nginx/ingress-nginx \ + --namespace $NAMESPACE \ + -f nginx-ingress-values.yaml +``` + +## Deploy a Sample App + +We'll need a sample app to test our configuration. I personally like the '[echoserver](https://github.com/cilium/echoserver)' app from the cilium team. It's nice, as it returns the HTTP headers as the web response, which can be very useful in certificate testing. + +```bash +cat <*Note:* In this walk through we did not add encryption between the ingress controller and the backend deployment. This can be done by sharing the same, or different, certificate to the deployment pods. You then enable backend encryption on the ingress controller. Alternatively, you could use a service mesh between the ingress and the backend deployment. diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint1.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint1.jpg new file mode 100644 index 0000000..bea9dec Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint1.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint2.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint2.jpg new file mode 100644 index 0000000..7b01aca Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint2.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint3.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint3.jpg new file mode 100644 index 0000000..9e31ba6 Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint3.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint4.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint4.jpg new file mode 100644 index 0000000..da24546 Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint4.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint5.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint5.jpg new file mode 100644 index 0000000..a30b763 Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/addendpoint5.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/afd-adddomain1.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/afd-adddomain1.jpg new file mode 100644 index 0000000..67f40a4 Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/afd-adddomain1.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/afd-adddomain2.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/afd-adddomain2.jpg new file mode 100644 index 0000000..0d141ab Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/afd-adddomain2.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/afd-adddomain3.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/afd-adddomain3.jpg new file mode 100644 index 0000000..5383f26 Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/afd-adddomain3.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/approvepl1.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/approvepl1.jpg new file mode 100644 index 0000000..eeb0cda Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/approvepl1.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/approvepl2.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/approvepl2.jpg new file mode 100644 index 0000000..06e9396 Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/approvepl2.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/approvepl3.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/approvepl3.jpg new file mode 100644 index 0000000..c4047cb Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/approvepl3.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/approvepl4.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/approvepl4.jpg new file mode 100644 index 0000000..833c7a4 Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/approvepl4.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/dnscname.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/dnscname.jpg new file mode 100644 index 0000000..a63ec38 Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/dnscname.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/linkcert.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/linkcert.jpg new file mode 100644 index 0000000..a3a633e Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/linkcert.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/origin-setup1.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/origin-setup1.jpg new file mode 100644 index 0000000..87c952a Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/origin-setup1.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/origin-setup2.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/origin-setup2.jpg new file mode 100644 index 0000000..e0c50cd Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/origin-setup2.jpg differ diff --git a/docs/assets/img/2024-11-05-afd-aks-ingress-tls/origin-setup3.jpg b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/origin-setup3.jpg new file mode 100644 index 0000000..dd87013 Binary files /dev/null and b/docs/assets/img/2024-11-05-afd-aks-ingress-tls/origin-setup3.jpg differ