diff --git a/.spelling b/.spelling index 24047110b9..7cd8fbf5a0 100644 --- a/.spelling +++ b/.spelling @@ -602,6 +602,16 @@ liveness apiservices arm64 IaC +Kyverno +Stakater +Reloader +reloader +yq +usr +ssl +cert.pem +Rollout +rollout # TEMPORARY # these are temporarily ignored because the spellchecker diff --git a/content/docs/manifest.json b/content/docs/manifest.json index 2c99b44c50..61b90ca1b4 100644 --- a/content/docs/manifest.json +++ b/content/docs/manifest.json @@ -428,6 +428,10 @@ { "title": "Securing Ingresses with ZeroSSL", "path": "/docs/tutorials/zerossl/zerossl.md" + }, + { + "title": "Managing public trust in kubernetes with trust-manager", + "path": "/docs/tutorials/getting-started-with-trust-manager/README.md" } ] }, diff --git a/content/docs/tutorials/getting-started-with-trust-manager/README.md b/content/docs/tutorials/getting-started-with-trust-manager/README.md new file mode 100644 index 0000000000..5ce09cbc69 --- /dev/null +++ b/content/docs/tutorials/getting-started-with-trust-manager/README.md @@ -0,0 +1,603 @@ +--- +title: Managing public trust in kubernetes with trust-manager +description: Learn how to deploy and configure trust-manager to automatically distribute your approved Public CA configuration to your Kubernetes cluster. +--- + +*Last Verified: 19 June 2023* + +In this tutorial we will walk through how we can use +[trust-manager](https://cert-manager.io/docs/projects/trust-manager/) to +distribute publicly trusted Certificate Authority (CA) certificates inside +a Kubernetes cluster. Once distributed we will also show: + +- How you can automatically reload applications when your trust bundle changes +- How you can enforce applications to use your distributed CA bundle + +From there we will use a simple `curl` pod to show how to automatically mount +the trusted CA `Bundle`, so it can be used without having to configure curl +manually. This mimics how an application would not need any additional +configuration to make use of your trusted CA certificates bundle. + +In this tutorial we will be limiting the scope of our changes to only impact +the `team-a` namespace. To get the most out of these features you will want +to remove this limitation. + +> **Note:** All resources provided are demonstrative and should be reviewed + properly before using in production environments. + +## Prerequisites + +**💻 Software** + +1. [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl): The Kubernetes +command-line tool which allows you to configure Kubernetes clusters. +1. [helm](https://helm.sh/): A package manager for Kubernetes. +1. [yq](https://github.com/mikefarah/yq#install): A command line tool for +parsing YAML with helpful coloring. + +## Distribute Public CA Trust + +Let us first setup trust-manager and have our public CAs distributed to our +demo namespace: `team-a`. + +### Setup Application & Bundle + +1) Install trust-manager following the + [instructions here](../../projects/trust-manager/README.md#installation). + +1) Create your first `Bundle` resource including only Public CA certificates + + ```yaml file=./trust/bundle-public.yaml + ``` + + ```shell + kubectl apply -f - < **Note**: this is to limit the scope of our trust bundle to only the + `team-a` namespace as mentioned previously. + +1) Verify that the trust-manager controller has correctly propagated the + CA bundle to the namespace: + + ```shell + kubectl get configmap -n team-a public-bundle -o yaml + ``` + + Note that this output should be quite long. This is because the default + public bundle that we use has a lot of public CAs in it. + +### Mount Trust Bundle to Application with Automatic Use + +To use our trusted CAs we will mount them to the application in a default +location that most applications expect to find a `ca-certificates.crt` file. +The benefit to this approach is that most application code inside the container +will use this file by default and not require any additional configuration. There is the added benefit that you will be mounting over the top of +`/etc/ssl/certs` which will remove existing CA certificates, usually +present from a container base image or pulled in during CI builds. + +> **WARNING:** We have chosen one well known location in this example which + is used by alpine and `curl` for sourcing trusted CAs. This is not the only + location that can be used, so a container may have other default locations. + If you want to see where default CAs are located you can use + [paranoia](https://github.com/jetstack/paranoia) to inspect a built container + image. + +1) Apply the application deployment: + + ```yaml file=./trust/deploy-auto.yaml + ``` + + ```shell + kubectl apply -f - < ..data/ca-certificates.crt + ``` + + Note that normally this container image the output would look something + like the following, when there is no volume overriding this directory: + + ``` + ~ $ ls -ltr /etc/ssl/certs/ + total 608 + -rw-r--r-- 1 root root 214222 Apr 14 01:11 ca-certificates.crt + lrwxrwxrwx 1 root root 52 Apr 14 01:11 ca-cert-vTrus_Root_CA.pem -> /usr/share/ca-certificates/mozilla/vTrus_Root_CA.crt + lrwxrwxrwx 1 root root 56 Apr 14 01:11 ca-cert-vTrus_ECC_Root_CA.pem -> /usr/share/ca-certificates/mozilla/vTrus_ECC_Root_CA.crt + ... + lrwxrwxrwx 1 root root 53 Apr 14 01:11 02265526.0 -> ca-cert-Entrust_Root_Certification_Authority_-_G2.pem + lrwxrwxrwx 1 root root 31 Apr 14 01:11 002c0b4f.0 -> ca-cert-GlobalSign_Root_R46.pem + ``` + +1) Make a HTTPS call out to a well known site to validate `curl` works without +having to pass the additional `--cacert` flag: + + ```shell + curl -v https://bbc.co.uk/news + ``` + + Success will result in a valid TLS connection such as: + + ``` + * Trying 151.101.0.81:443... + * Connected to bbc.co.uk (151.101.0.81) port 443 (#0) + * ALPN: offers h2,http/1.1 + * TLSv1.3 (OUT), TLS handshake, Client hello (1): + * CAfile: /etc/ssl/certs/ca-certificates.crt + * CApath: none + * TLSv1.3 (IN), TLS handshake, Server hello (2): + * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): + * TLSv1.3 (IN), TLS handshake, Certificate (11): + * TLSv1.3 (IN), TLS handshake, CERT verify (15): + * TLSv1.3 (IN), TLS handshake, Finished (20): + * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): + * TLSv1.3 (OUT), TLS handshake, Finished (20): + * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 + * ALPN: server accepted h2 + * Server certificate: + * subject: C=GB; ST=London; L=London; O=BRITISH BROADCASTING CORPORATION; CN=www.bbc.com + * start date: Mar 14 06:16:13 2023 GMT + * expire date: Apr 14 06:16:12 2024 GMT + * subjectAltName: host "bbc.co.uk" matched cert's "bbc.co.uk" + * issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign RSA OV SSL CA 2018 + * SSL certificate verify ok. + ``` + +1. Exit the container: `exit` + +## Configure Real Applications + +Based on the example above, Kubernetes is able to mount over the top of the +default CA certificate bundle. You can use this with applications assuming you +know where the default locations they retrieve CA certificates from. + +For example with `Go` your application is configurable with either +`SSL_CERT_FILE` or `SSL_CERT_DIR` to point to the default CA certificate +file location. + +See more details [here](https://go.dev/src/crypto/x509/root_unix.go) and +for the default locations on various OS bases, check +[here](https://go.dev/src/crypto/x509/root_linux.go) + +```go +// Possible certificate files; stop after finding one. +var certFiles = []string{ + "/etc/ssl/certs/ca-certificates.crt", // Debian/Ubuntu/Gentoo etc. + "/etc/pki/tls/certs/ca-bundle.crt", // Fedora/RHEL 6 + "/etc/ssl/ca-bundle.pem", // OpenSUSE + "/etc/pki/tls/cacert.pem", // OpenELEC + "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", // CentOS/RHEL 7 + "/etc/ssl/cert.pem", // Alpine Linux +} + +// Possible directories with certificate files; all will be read. +var certDirectories = []string{ + "/etc/ssl/certs", // SLES10/SLES11, https://golang.org/issue/12139 + "/etc/pki/tls/certs", // Fedora/RHEL + "/ +``` + +Having checked Python the `ssl` library uses the same two environment variables +for finding the trusted CAs: `SSL_CERT_DIR` and / or `SSL_CERT_FILE`. You can +verify this [in documentation](https://docs.python.org/3/library/ssl.html#ssl.get_default_verify_paths) +and from a `python3` runtime: + +```python3 +>>> import ssl +>>> ssl.get_default_verify_paths() +DefaultVerifyPaths(cafile=None, capath='/usr/lib/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs') +``` + +This should mean that any CAs mounted in a file and any of the following files +will be trusted by any python application runtime, similar to `Go`: + +- '/usr/lib/ssl/cert.pem' +- '/usr/lib/ssl/certs/*' + +Similar could be achieved with other languages. + +## Automate and Enforce + +So now we have mounted trust-manager's bundle manually, you might be thinking: + +- What happens if the CA Bundle is changed, how do I get that change to my + application? +- How do I ensure that my CA Bundle is mounted to all applications in my + cluster without having to request changes from my tenants? + +Let's tackle both of these scenarios using additional Open Source tools. + +### Rollout CA Bundle Changes + +If your CA bundle changes, those changes will be synced to the namespaces +pretty quickly. This change will be reflected in the volume attached to the +container, but most applications will not pickup on the file system change. +The common approach is restarting the client application deployment, through +the use of `kubectl rollout restart deployment `. There is an +option to automate this process through a third party piece of open-source +software. + +Using [Stakater Reloader](https://github.com/stakater/Reloader) it is +possible to reload or rollout a deployment whenever a `ConfigMap` or `Secret` +changes. So whenever the `Bundle`'s target is synced, the Reloader component +can pick up this change and rollout applications mounting those resource +as volumes or environment variables. + +**Please note** that there are many alternative pieces of software that you +could bundle or write into your application container. They would simply watch +the file system for changes and trigger a reload of the application process. +Such an approach requires container image or code changes and this could be +difficult to implement with many tenants. The advantage to using reloader here +is that it's a generic solution, applicable to all applications running in a +cluster. + +1. Continuing with the reloader, it can be installed with helm: + + ```shell + helm repo add stakater https://stakater.github.io/stakater-charts + helm repo update + helm install reloader stakater/reloader -n stakater-reloader --create-namespace --set fullnameOverride=reloader + ``` + +1. We can reuse the deployment `sleep-auto` from the previous section and + configured it to enabled the reload functionality: + + ```shell + kubectl annotate deployment -n team-a sleep-auto reloader.stakater.com/auto="true" + ``` + + **Please note** there are several configuration options to configure + the reloader tooling and this is only the most basic example. Refer to + [the documentation](https://github.com/stakater/Reloader#how-to-use-reloader) + for more detailed examples. + +1. In another terminal watch the application rollout: + + ```shell + kubectl get po -n team-a -w + ``` + +1. To test this change we can edit our `Bundle` resource to remove all the + default Public CA certificates and only provide one CA certificate instead: + + ```yaml file=./trust/bundle-one-ca.yaml + ``` + + ```shell + kubectl apply -f - <