From ac426f43157bcbbfdb07320a130b75e04da5b62e Mon Sep 17 00:00:00 2001 From: Steve Griffith Date: Thu, 9 Nov 2023 12:18:19 -0500 Subject: [PATCH] added two new posts on image signing and verification --- .../_posts/2023-11-09-part1-notation-usage.md | 241 ++++++++++++++++++ ...2023-11-09-part2-aks-image-verification.md | 143 +++++++++++ 2 files changed, 384 insertions(+) create mode 100644 docs/_posts/2023-11-09-part1-notation-usage.md create mode 100644 docs/_posts/2023-11-09-part2-aks-image-verification.md diff --git a/docs/_posts/2023-11-09-part1-notation-usage.md b/docs/_posts/2023-11-09-part1-notation-usage.md new file mode 100644 index 0000000..d9b8a7b --- /dev/null +++ b/docs/_posts/2023-11-09-part1-notation-usage.md @@ -0,0 +1,241 @@ +--- +title: Image Verification Part 1 - Notation CLI +description: How to use the Notary project's 'notation' cli tool to sign and verify container images +authors: + - steve_griffith +--- + +# Part 1 - Image Signing with Notation + +## Introduction + +This is part one of a two part post on container image signing and runtime verification. In this post, we'll walk through the [notation](https://github.com/notaryproject/notation) project and its ability to sign container images, using the [Notary](https://github.com/notaryproject/specifications) project specification. In the next post, we'll walk through setting up [gatekeeper](https://open-policy-agent.github.io/gatekeeper/website/) and [ratify](https://github.com/deislabs/ratify/blob/main/README.md) to perform policy based runtime verification of images. + +## Installation + +For this walk through you'll need to have Docker running locally, and have the notation cli installed. You can find the install steps for each in the list below: + +* [Docker](https://www.docker.com/get-started/) +* [Notation CLI](https://notaryproject.dev/docs/user-guides/installation/cli/) + +## Azure Container Registry + +You can really use any OCI compatible registry, but since we'll later be using AKS for verification, we'll create an Azure Container Registery for use in the following steps. + +```bash +# Setup Environment Variables +RG=EphNotationTesting +LOC=eastus +ACR_NAME=mynotationlab + +# Create the resource group +az group create -n $RG -l $LOC + +# Create the Azure Container Registry +az acr create -g $RG -n $ACR_NAME --sku Standard + +# Login to the ACR +az acr login -n $ACR_NAME +``` + +We could build our own container image from scratch, but the image signing process doesnt really change, so let's just import an existing image to use. This is a good demonstration of the best practice of never directly pulling public images. Instead of direclty pulling public images into your cluster, you should import and verify any public images you wish to use. We'll take it a step further by also signing the imported image. + +```bash +# For later testing, lets also import an unsigned image +az acr import --name $ACR_NAME --source docker.io/library/nginx:1.25.3 --image nginx:1.25.3 +``` + +Now we can play around with notation. Notation does require you use the image SHA instead of tags, so we'll get that first. + +```bash +# Get the nginx image SHA +IMAGE_SHA=$(az acr repository show -n $ACR_NAME --image "nginx:1.25.3" -o tsv --query digest) +``` + +Now we'll use the notation cli to check the image for existing signatures and then create and apply a new signature. The notation project does provide the ability to generate a test certificate locally to get you up and running quickly. We'll try that first and then move on to using Azure Key Vault for the certificate. + +```bash +# List the signatures on the image +# You should see the image has no signatures +notation ls $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA + +# Generate a test RSA key and self-signed certificat +notation cert generate-test --default "brooklyn.io" + +# List certs to confirm the cert exists +notation cert ls + +# Sign the image +notation sign $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA + +# Now check the image signatures +# You should now see that the image is signed +notation ls $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA +``` + +You should now have a successfully signed image. Next, we can use notation to verify the image has been signed. For that we'll need to create a local trust policy for notation. + +```bash +# Create a trust policy for notation +cat < ./trustpolicy.json +{ + "version": "1.0", + "trustPolicies": [ + { + "name": "brooklyn-images", + "registryScopes": [ "*" ], + "signatureVerification": { + "level" : "strict" + }, + "trustStores": [ "ca:brooklyn.io" ], + "trustedIdentities": [ + "*" + ] + } + ] +} +EOF + +# Import the policy +notation policy import ./trustpolicy.json + +# Show the policy +notation policy show + +# Verify the image meets the policy +# You should get a message that the signature was verified +notation verify $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA +``` + +## Cleanup + +Before moving on, we should delete and re-import the image, since we dont want the local test certificate on the image any more. You can also remove the signature with [oras](https://notaryproject.dev/docs/user-guides/how-to/manage-signatures/#delete-a-signature-on-an-artifact), but we'll keep it simple for now. + +```bash +# Delete and re-import the image +az acr repository delete -n $ACR_NAME --repository nginx -y +az acr import --name $ACR_NAME --source docker.io/library/nginx:1.25.3 --image nginx:1.25.3 +IMAGE_SHA=$(az acr repository show -n $ACR_NAME --image "nginx:1.25.3" -o tsv --query digest) +``` + +## Sign Images with a Cert from Azure Key Vault + +Using the notation locally generated test certificate does give you some level of comfort about the source of the image, but you really want to use a certificate from a trusted store. It's also important to have an accessible store that can be used by remote processes, like your devops pipeline. Fortunately, notation provides a plugin for Azure Key Vault which can be used to sign images with certifiates source from Azure Key Vault. + +First you need to install the Azure Key Vault plug-in for notation. You can find instructions for your OS [here](https://github.com/Azure/notation-azure-kv#installation-the-akv-plugin). + +```bash +# Confirm you successfuly installed the plugin +notation plugin list + +# Sample Output +NAME DESCRIPTION VERSION CAPABILITIES ERROR +azure-kv Notation Azure Key Vault plugin 1.0.1 [SIGNATURE_GENERATOR.RAW] +``` + +Now we'll create an Azure Key Vault instance for our test and configure the signature. It's important that the secret properties specify that the content type is 'application/x-pem-file' as ratify, which we'll use later, cannot read the default PFX format. + +```bash +AKV_NAME=mynotationtest + +# Create the key vault +az keyvault create --name $AKV_NAME --resource-group $RG + +# Set some variables for the cert creation +# Name of the certificate created in AKV +CERT_NAME=brooklyn-io +CERT_SUBJECT="CN=brooklyn.io,O=Notation,L=Brooklyn,ST=NY,C=US" +CERT_PATH=./${CERT_NAME}.pem + +# Set the access policy for yourself to create and get certs +USER_ID=$(az ad signed-in-user show --query id -o tsv) +az keyvault set-policy -n $AKV_NAME --certificate-permissions create get --key-permissions sign --object-id $USER_ID + +# Create the Key Vault certificate policy file +cat < ./brooklyn_io_policy.json +{ + "issuerParameters": { + "certificateTransparency": null, + "name": "Self" + }, + "keyProperties": { + "exportable": false, + "keySize": 2048, + "keyType": "RSA", + "reuseKey": true + }, + "secretProperties": { + "contentType": "application/x-pem-file" + }, + "x509CertificateProperties": { + "ekus": [ + "1.3.6.1.5.5.7.3.3" + ], + "keyUsage": [ + "digitalSignature" + ], + "subject": "CN=brooklyn.io,O=Notation,L=Brooklyn,ST=NY,C=US", + "validityInMonths": 12 + } +} +EOF + +# Create the signing certificate +az keyvault certificate create -n $CERT_NAME --vault-name $AKV_NAME -p @brooklyn_io_policy.json + +# Get the Key ID of the signing key +KEY_ID=$(az keyvault certificate show -n $CERT_NAME --vault-name $AKV_NAME --query 'kid' -o tsv) + +# Now sign the previosly imported nginx image +# You should get a confirmation that the image was successfully signed +notation sign --signature-format cose --id $KEY_ID --plugin azure-kv --plugin-config self_signed=true $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA + +# Confirm the signature +notation ls $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA +``` + +Finally, now that we have the image signed with our certifcate from Azure Key Vault, lets set up our local environment trust policy to allow us to verify the image signature is valid locally. We'll need to download the certificate and add a local trust store. + +```bash +# Download the cert from Azure Key Vault so we can verify the image locally with the AKV cert +az keyvault certificate download --name $CERT_NAME --vault-name $AKV_NAME --file $CERT_PATH + +STORE_TYPE="ca" +STORE_NAME="brooklyn.io" +notation cert add --type $STORE_TYPE --store $STORE_NAME $CERT_PATH + +cat < ./trustpolicy.json +{ + "version": "1.0", + "trustPolicies": [ + { + "name": "brooklyn-images", + "registryScopes": [ "$ACR_NAME.azurecr.io/nginx" ], + "signatureVerification": { + "level" : "strict" + }, + "trustStores": [ "$STORE_TYPE:$STORE_NAME" ], + "trustedIdentities": [ + "x509.subject: $CERT_SUBJECT" + ] + } + ] +} +EOF + +# Import the policy and show it +notation policy import ./trustpolicy.json +notation policy show + +# Test image verification +notation verify $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA + +# Sample success message!!! +Successfully verified signature for mynotationlab.azurecr.io/nginx@sha256:86e53c4c16a6a276b204b0fd3a8143d86547c967dc8258b3d47c3a21bb68d3c6 +``` + +## Conclusion + +You should now have some familiarity with the notation cli tool and how to use it to sign container images in a container registry with both self-signed certificates and certificates from Azure Key Vault. The next step would be to enable verification in kubernetes. + +**Next:** [Part 2 - Image Verification in AKS](./part2-aks-image-verification.html) \ No newline at end of file diff --git a/docs/_posts/2023-11-09-part2-aks-image-verification.md b/docs/_posts/2023-11-09-part2-aks-image-verification.md new file mode 100644 index 0000000..a96d299 --- /dev/null +++ b/docs/_posts/2023-11-09-part2-aks-image-verification.md @@ -0,0 +1,143 @@ +--- +title: Image Verification Part 2 - Image Verification with Gatekeeper and Ratify +description: How to manually setup Gatekeeper and Ratify on an AKS cluster to enable runtime image signature verification. +authors: + - steve_griffith +--- + +# Part 2 - Image Verification with Gatekeeper and Ratify + +## Introduction + +In the prior post, we ran through using the notation cli tool to sign images in Azure Container Registry. If you havent gone through that post, I recommend you start there at [Part 1 - Image Signing with Notation](./part1-notation-usage.html) + +In this post, we'll walk through the steps to manually configure AKS with Gatekeeper and the Ratify project to enforce an image signature verification policy. + +## Cluster Creation and Setup + +For Ratify to work with Gatekeeper, we'll need a cluster with both the OIDC Issuer and Workload Idenitty add-ons enabled. In the last step below, we'll grab the OIDC Issuer URL for the cluster, which is used when you want to federate a Kubernetes Service Account with an Azure Active Directory Identity. + +```bash +# Set environment variables +RG=EphNotationTesting +LOC=eastus +ACR_NAME=mynotationlab +CLUSTER_NAME=imagesigninglab + +# Create the AKS Cluster +az aks create -g $RG -n $CLUSTER_NAME \ +--attach-acr $ACR_NAME \ +--enable-oidc-issuer \ +--enable-workload-identity + +# Get the cluster credentials +az aks get-credentials -g $RG -n $CLUSTER_NAME + +# Get the OIDC Issuer URL +export AKS_OIDC_ISSUER="$(az aks show -n ${CLUSTER_NAME} -g ${RG} --query "oidcIssuerProfile.issuerUrl" -otsv)" +``` + +## Managed Identity Setup + +Ratify will need to be able to read from the Key Vault, so we'll need to create a managed identity and grant it the proper rights on Azure Container Registry. The only right needed is 'acrpull', as ratify will pull the image to check it's signature. + +```bash +SUBSCRIPTION=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx +TENANT_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx +IDENTITY_NAME=ratify-identity +RATIFY_NAMESPACE=gatekeeper-system + +# Create the ratify identity +az identity create --name "${IDENTITY_NAME}" --resource-group "${RG}" --location "${LOC}" --subscription "${SUBSCRIPTION}" + +# Get the identity IDs +export IDENTITY_OBJECT_ID="$(az identity show --name "${IDENTITY_NAME}" --resource-group "${RG}" --query 'principalId' -otsv)" +export IDENTITY_CLIENT_ID=$(az identity show --name ${IDENTITY_NAME} --resource-group ${RG} --query 'clientId' -o tsv) + +# Grant the ratify identity acr pull rights +az role assignment create \ +--assignee-object-id ${IDENTITY_OBJECT_ID} \ +--role acrpull \ +--scope subscriptions/${SUBSCRIPTION}/resourceGroups/${RG}/providers/Microsoft.ContainerRegistry/registries/${ACR_NAME} + +# Federate the managed identity to the service account used by ratify +az identity federated-credential create \ +--name ratify-federated-credential \ +--identity-name "${IDENTITY_NAME}" \ +--resource-group "${RG}" \ +--issuer "${AKS_OIDC_ISSUER}" \ +--subject system:serviceaccount:"${RATIFY_NAMESPACE}":"ratify-admin" +``` + +Since Ratify will be checking the signature, it also needs the ability to get the secret from Azure Key Vault. + +```bash +# Grant the ratify identity rights +az keyvault set-policy --name ${AKV_NAME} \ +--secret-permissions get \ +--object-id ${IDENTITY_OBJECT_ID} +``` + +## Install Gatekeeper and Ratify + +While AKS does, now in preview, have a [managed add-on for gatekeeper and ratify](https://learn.microsoft.com/en-us/azure/aks/image-integrity?tabs=azure-cli), it's still in preview. For this post, we'll manually install both Gatekeeper and Ratify so that we can see all the moving parts and more easily debug any issues. + +```bash +# Install Gatekeeper +helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts + +helm install gatekeeper/gatekeeper \ + --name-template=gatekeeper \ + --namespace ${RATIFY_NAMESPACE} --create-namespace \ + --set enableExternalData=true \ + --set validatingWebhookTimeoutSeconds=5 \ + --set mutatingWebhookTimeoutSeconds=2 + +# Get the key vault URI which ratify will need +export VAULT_URI=$(az keyvault show --name ${AKV_NAME} --resource-group ${RG} --query "properties.vaultUri" -otsv) + +# Install Ratify +helm repo add ratify https://deislabs.github.io/ratify + +helm install ratify \ + ratify/ratify --atomic \ + --namespace ${RATIFY_NAMESPACE} --create-namespace \ + --set featureFlags.RATIFY_CERT_ROTATION=true \ + --set akvCertConfig.enabled=true \ + --set akvCertConfig.vaultURI=${VAULT_URI} \ + --set akvCertConfig.cert1Name=${CERT_NAME} \ + --set akvCertConfig.tenantId=${TENANT_ID} \ + --set oras.authProviders.azureWorkloadIdentityEnabled=true \ + --set azureWorkloadIdentity.clientId=${IDENTITY_CLIENT_ID} +``` + +Now that gatekeeper and ratify are running, lets apply a new constraint and policy template for the image verification policy. You should inspect the two files in the commands below for your own knowledge of how they work. + +```bash +# Create the gatekeeper policy template +kubectl apply -f ratify-policy-template.yaml + +# Apply the policy with a gatekeeper constraint +kubectl apply -f ratify-policy-constraint.yaml +``` + +## Test the policy! + +Our setup is complete. We can now try to create a pod using an unsigned and signed container image. + +```bash +# First try to use the docker hub nginx image, which is unsigned +# This should fail +kubectl run demo --namespace default --image=nginx:latest + +# Sample Error Message +Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Subject failed verification: docker.io/library/nginx@sha256:86e53c4c16a6a276b204b0fd3a8143d86547c967dc8258b3d47c3a21bb68d3c6 + +# Now try using our container image +# This pod should be successfully created! +kubectl run demo --namespace default --image=$ACR_NAME.azurecr.io/nginx@$IMAGE_SHA +``` + +## Conclusion + +Between this post and [Part 1](./part1-notation-usage.html), we learned about the notation cli tool, which can be used to sign container images via the notary specification. We signed images with both a local test certificate, as well as a certificate managed by Azure Key Vault. Finally, we enabled Gatekeeper and Ratify on an AKS cluster to provide an image signature verification policy. \ No newline at end of file