Skip to content

Commit

Permalink
Merge pull request #7 from appdevgbb/sg-local-dev
Browse files Browse the repository at this point in the history 
added two new posts on image signing and verification
  • Loading branch information
@swgriffith
swgriffith authored Nov 9, 2023
2 parents 68cbd2e + ac426f4 commit dd4f72b
Show file tree
Hide file tree
Showing 2 changed files with 384 additions and 0 deletions.
241 changes: 241 additions & 0 deletions docs/_posts/2023-11-09-part1-notation-usage.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,241 @@
---
title: Image Verification Part 1 - Notation CLI
description: How to use the Notary project's 'notation' cli tool to sign and verify container images
authors:
- steve_griffith
---

# Part 1 - Image Signing with Notation

## Introduction

This is part one of a two part post on container image signing and runtime verification. In this post, we'll walk through the [notation](https://github.com/notaryproject/notation) project and its ability to sign container images, using the [Notary](https://github.com/notaryproject/specifications) project specification. In the next post, we'll walk through setting up [gatekeeper](https://open-policy-agent.github.io/gatekeeper/website/) and [ratify](https://github.com/deislabs/ratify/blob/main/README.md) to perform policy based runtime verification of images.

## Installation

For this walk through you'll need to have Docker running locally, and have the notation cli installed. You can find the install steps for each in the list below:

* [Docker](https://www.docker.com/get-started/)
* [Notation CLI](https://notaryproject.dev/docs/user-guides/installation/cli/)

## Azure Container Registry

You can really use any OCI compatible registry, but since we'll later be using AKS for verification, we'll create an Azure Container Registery for use in the following steps.

```bash
# Setup Environment Variables
RG=EphNotationTesting
LOC=eastus
ACR_NAME=mynotationlab

# Create the resource group
az group create -n $RG -l $LOC

# Create the Azure Container Registry
az acr create -g $RG -n $ACR_NAME --sku Standard

# Login to the ACR
az acr login -n $ACR_NAME
```

We could build our own container image from scratch, but the image signing process doesnt really change, so let's just import an existing image to use. This is a good demonstration of the best practice of never directly pulling public images. Instead of direclty pulling public images into your cluster, you should import and verify any public images you wish to use. We'll take it a step further by also signing the imported image.

```bash
# For later testing, lets also import an unsigned image
az acr import --name $ACR_NAME --source docker.io/library/nginx:1.25.3 --image nginx:1.25.3
```

Now we can play around with notation. Notation does require you use the image SHA instead of tags, so we'll get that first.

```bash
# Get the nginx image SHA
IMAGE_SHA=$(az acr repository show -n $ACR_NAME --image "nginx:1.25.3" -o tsv --query digest)
```

Now we'll use the notation cli to check the image for existing signatures and then create and apply a new signature. The notation project does provide the ability to generate a test certificate locally to get you up and running quickly. We'll try that first and then move on to using Azure Key Vault for the certificate.

```bash
# List the signatures on the image
# You should see the image has no signatures
notation ls $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA

# Generate a test RSA key and self-signed certificat
notation cert generate-test --default "brooklyn.io"

# List certs to confirm the cert exists
notation cert ls

# Sign the image
notation sign $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA

# Now check the image signatures
# You should now see that the image is signed
notation ls $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA
```

You should now have a successfully signed image. Next, we can use notation to verify the image has been signed. For that we'll need to create a local trust policy for notation.

```bash
# Create a trust policy for notation
cat <<EOF > ./trustpolicy.json
{
"version": "1.0",
"trustPolicies": [
{
"name": "brooklyn-images",
"registryScopes": [ "*" ],
"signatureVerification": {
"level" : "strict"
},
"trustStores": [ "ca:brooklyn.io" ],
"trustedIdentities": [
"*"
]
}
]
}
EOF

# Import the policy
notation policy import ./trustpolicy.json

# Show the policy
notation policy show

# Verify the image meets the policy
# You should get a message that the signature was verified
notation verify $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA
```

## Cleanup

Before moving on, we should delete and re-import the image, since we dont want the local test certificate on the image any more. You can also remove the signature with [oras](https://notaryproject.dev/docs/user-guides/how-to/manage-signatures/#delete-a-signature-on-an-artifact), but we'll keep it simple for now.

```bash
# Delete and re-import the image
az acr repository delete -n $ACR_NAME --repository nginx -y
az acr import --name $ACR_NAME --source docker.io/library/nginx:1.25.3 --image nginx:1.25.3
IMAGE_SHA=$(az acr repository show -n $ACR_NAME --image "nginx:1.25.3" -o tsv --query digest)
```

## Sign Images with a Cert from Azure Key Vault

Using the notation locally generated test certificate does give you some level of comfort about the source of the image, but you really want to use a certificate from a trusted store. It's also important to have an accessible store that can be used by remote processes, like your devops pipeline. Fortunately, notation provides a plugin for Azure Key Vault which can be used to sign images with certifiates source from Azure Key Vault.

First you need to install the Azure Key Vault plug-in for notation. You can find instructions for your OS [here](https://github.com/Azure/notation-azure-kv#installation-the-akv-plugin).

```bash
# Confirm you successfuly installed the plugin
notation plugin list

# Sample Output
NAME DESCRIPTION VERSION CAPABILITIES ERROR
azure-kv Notation Azure Key Vault plugin 1.0.1 [SIGNATURE_GENERATOR.RAW] <nil>
```

Now we'll create an Azure Key Vault instance for our test and configure the signature. It's important that the secret properties specify that the content type is 'application/x-pem-file' as ratify, which we'll use later, cannot read the default PFX format.

```bash
AKV_NAME=mynotationtest

# Create the key vault
az keyvault create --name $AKV_NAME --resource-group $RG

# Set some variables for the cert creation
# Name of the certificate created in AKV
CERT_NAME=brooklyn-io
CERT_SUBJECT="CN=brooklyn.io,O=Notation,L=Brooklyn,ST=NY,C=US"
CERT_PATH=./${CERT_NAME}.pem

# Set the access policy for yourself to create and get certs
USER_ID=$(az ad signed-in-user show --query id -o tsv)
az keyvault set-policy -n $AKV_NAME --certificate-permissions create get --key-permissions sign --object-id $USER_ID

# Create the Key Vault certificate policy file
cat <<EOF > ./brooklyn_io_policy.json
{
"issuerParameters": {
"certificateTransparency": null,
"name": "Self"
},
"keyProperties": {
"exportable": false,
"keySize": 2048,
"keyType": "RSA",
"reuseKey": true
},
"secretProperties": {
"contentType": "application/x-pem-file"
},
"x509CertificateProperties": {
"ekus": [
"1.3.6.1.5.5.7.3.3"
],
"keyUsage": [
"digitalSignature"
],
"subject": "CN=brooklyn.io,O=Notation,L=Brooklyn,ST=NY,C=US",
"validityInMonths": 12
}
}
EOF

# Create the signing certificate
az keyvault certificate create -n $CERT_NAME --vault-name $AKV_NAME -p @brooklyn_io_policy.json

# Get the Key ID of the signing key
KEY_ID=$(az keyvault certificate show -n $CERT_NAME --vault-name $AKV_NAME --query 'kid' -o tsv)

# Now sign the previosly imported nginx image
# You should get a confirmation that the image was successfully signed
notation sign --signature-format cose --id $KEY_ID --plugin azure-kv --plugin-config self_signed=true $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA

# Confirm the signature
notation ls $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA
```

Finally, now that we have the image signed with our certifcate from Azure Key Vault, lets set up our local environment trust policy to allow us to verify the image signature is valid locally. We'll need to download the certificate and add a local trust store.

```bash
# Download the cert from Azure Key Vault so we can verify the image locally with the AKV cert
az keyvault certificate download --name $CERT_NAME --vault-name $AKV_NAME --file $CERT_PATH

STORE_TYPE="ca"
STORE_NAME="brooklyn.io"
notation cert add --type $STORE_TYPE --store $STORE_NAME $CERT_PATH

cat <<EOF > ./trustpolicy.json
{
"version": "1.0",
"trustPolicies": [
{
"name": "brooklyn-images",
"registryScopes": [ "$ACR_NAME.azurecr.io/nginx" ],
"signatureVerification": {
"level" : "strict"
},
"trustStores": [ "$STORE_TYPE:$STORE_NAME" ],
"trustedIdentities": [
"x509.subject: $CERT_SUBJECT"
]
}
]
}
EOF

# Import the policy and show it
notation policy import ./trustpolicy.json
notation policy show

# Test image verification
notation verify $ACR_NAME.azurecr.io/nginx@$IMAGE_SHA

# Sample success message!!!
Successfully verified signature for mynotationlab.azurecr.io/nginx@sha256:86e53c4c16a6a276b204b0fd3a8143d86547c967dc8258b3d47c3a21bb68d3c6
```

## Conclusion

You should now have some familiarity with the notation cli tool and how to use it to sign container images in a container registry with both self-signed certificates and certificates from Azure Key Vault. The next step would be to enable verification in kubernetes.

**Next:** [Part 2 - Image Verification in AKS](./part2-aks-image-verification.html)
143 changes: 143 additions & 0 deletions docs/_posts/2023-11-09-part2-aks-image-verification.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
---
title: Image Verification Part 2 - Image Verification with Gatekeeper and Ratify
description: How to manually setup Gatekeeper and Ratify on an AKS cluster to enable runtime image signature verification.
authors:
- steve_griffith
---

# Part 2 - Image Verification with Gatekeeper and Ratify

## Introduction

In the prior post, we ran through using the notation cli tool to sign images in Azure Container Registry. If you havent gone through that post, I recommend you start there at [Part 1 - Image Signing with Notation](./part1-notation-usage.html)

In this post, we'll walk through the steps to manually configure AKS with Gatekeeper and the Ratify project to enforce an image signature verification policy.

## Cluster Creation and Setup

For Ratify to work with Gatekeeper, we'll need a cluster with both the OIDC Issuer and Workload Idenitty add-ons enabled. In the last step below, we'll grab the OIDC Issuer URL for the cluster, which is used when you want to federate a Kubernetes Service Account with an Azure Active Directory Identity.

```bash
# Set environment variables
RG=EphNotationTesting
LOC=eastus
ACR_NAME=mynotationlab
CLUSTER_NAME=imagesigninglab

# Create the AKS Cluster
az aks create -g $RG -n $CLUSTER_NAME \
--attach-acr $ACR_NAME \
--enable-oidc-issuer \
--enable-workload-identity

# Get the cluster credentials
az aks get-credentials -g $RG -n $CLUSTER_NAME

# Get the OIDC Issuer URL
export AKS_OIDC_ISSUER="$(az aks show -n ${CLUSTER_NAME} -g ${RG} --query "oidcIssuerProfile.issuerUrl" -otsv)"
```

## Managed Identity Setup

Ratify will need to be able to read from the Key Vault, so we'll need to create a managed identity and grant it the proper rights on Azure Container Registry. The only right needed is 'acrpull', as ratify will pull the image to check it's signature.

```bash
SUBSCRIPTION=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
TENANT_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx
IDENTITY_NAME=ratify-identity
RATIFY_NAMESPACE=gatekeeper-system

# Create the ratify identity
az identity create --name "${IDENTITY_NAME}" --resource-group "${RG}" --location "${LOC}" --subscription "${SUBSCRIPTION}"

# Get the identity IDs
export IDENTITY_OBJECT_ID="$(az identity show --name "${IDENTITY_NAME}" --resource-group "${RG}" --query 'principalId' -otsv)"
export IDENTITY_CLIENT_ID=$(az identity show --name ${IDENTITY_NAME} --resource-group ${RG} --query 'clientId' -o tsv)

# Grant the ratify identity acr pull rights
az role assignment create \
--assignee-object-id ${IDENTITY_OBJECT_ID} \
--role acrpull \
--scope subscriptions/${SUBSCRIPTION}/resourceGroups/${RG}/providers/Microsoft.ContainerRegistry/registries/${ACR_NAME}

# Federate the managed identity to the service account used by ratify
az identity federated-credential create \
--name ratify-federated-credential \
--identity-name "${IDENTITY_NAME}" \
--resource-group "${RG}" \
--issuer "${AKS_OIDC_ISSUER}" \
--subject system:serviceaccount:"${RATIFY_NAMESPACE}":"ratify-admin"
```

Since Ratify will be checking the signature, it also needs the ability to get the secret from Azure Key Vault.

```bash
# Grant the ratify identity rights
az keyvault set-policy --name ${AKV_NAME} \
--secret-permissions get \
--object-id ${IDENTITY_OBJECT_ID}
```

## Install Gatekeeper and Ratify

While AKS does, now in preview, have a [managed add-on for gatekeeper and ratify](https://learn.microsoft.com/en-us/azure/aks/image-integrity?tabs=azure-cli), it's still in preview. For this post, we'll manually install both Gatekeeper and Ratify so that we can see all the moving parts and more easily debug any issues.

```bash
# Install Gatekeeper
helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts

helm install gatekeeper/gatekeeper \
--name-template=gatekeeper \
--namespace ${RATIFY_NAMESPACE} --create-namespace \
--set enableExternalData=true \
--set validatingWebhookTimeoutSeconds=5 \
--set mutatingWebhookTimeoutSeconds=2

# Get the key vault URI which ratify will need
export VAULT_URI=$(az keyvault show --name ${AKV_NAME} --resource-group ${RG} --query "properties.vaultUri" -otsv)

# Install Ratify
helm repo add ratify https://deislabs.github.io/ratify

helm install ratify \
ratify/ratify --atomic \
--namespace ${RATIFY_NAMESPACE} --create-namespace \
--set featureFlags.RATIFY_CERT_ROTATION=true \
--set akvCertConfig.enabled=true \
--set akvCertConfig.vaultURI=${VAULT_URI} \
--set akvCertConfig.cert1Name=${CERT_NAME} \
--set akvCertConfig.tenantId=${TENANT_ID} \
--set oras.authProviders.azureWorkloadIdentityEnabled=true \
--set azureWorkloadIdentity.clientId=${IDENTITY_CLIENT_ID}
```

Now that gatekeeper and ratify are running, lets apply a new constraint and policy template for the image verification policy. You should inspect the two files in the commands below for your own knowledge of how they work.

```bash
# Create the gatekeeper policy template
kubectl apply -f ratify-policy-template.yaml

# Apply the policy with a gatekeeper constraint
kubectl apply -f ratify-policy-constraint.yaml
```

## Test the policy!

Our setup is complete. We can now try to create a pod using an unsigned and signed container image.

```bash
# First try to use the docker hub nginx image, which is unsigned
# This should fail
kubectl run demo --namespace default --image=nginx:latest

# Sample Error Message
Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [ratify-constraint] Subject failed verification: docker.io/library/nginx@sha256:86e53c4c16a6a276b204b0fd3a8143d86547c967dc8258b3d47c3a21bb68d3c6

# Now try using our container image
# This pod should be successfully created!
kubectl run demo --namespace default --image=$ACR_NAME.azurecr.io/nginx@$IMAGE_SHA
```

## Conclusion

Between this post and [Part 1](./part1-notation-usage.html), we learned about the notation cli tool, which can be used to sign container images via the notary specification. We signed images with both a local test certificate, as well as a certificate managed by Azure Key Vault. Finally, we enabled Gatekeeper and Ratify on an AKS cluster to provide an image signature verification policy.

0 comments on commit dd4f72b

Please sign in to comment.