feat: Add support for mutual TLS on kubewarden controller

[viccuad]

Description

Part of #630.

For kubewarden-controller chart, add a new .Values.mTLS section, where one can enable mutual TLS on the controller image.

The controller expects a ConfigMap with a Certificate Authority that is valid in the Kubernetes API server.

It will use this CA to create client certificates and keys that will be used by the policy-servers, so they can communicate via mTLS with the API server.

Test

TODO

Additional Information

Tradeoff

Potential improvement

[kravciak]

How to setup k3d with mTLS

Create certificates:

FQDN=mtls.kubewarden.io

# Create CA
openssl req -nodes -batch -x509 -sha256 -days 365 -newkey rsa:2048 -keyout rootCA.key -out rootCA.crt
# Create CSR
openssl req -nodes -batch -newkey rsa:2048 -keyout domain.key -out domain.csr \
    -addext "subjectAltName = DNS:$FQDN"
# Create CRT
openssl x509 -req -CA rootCA.crt -CAkey rootCA.key -in domain.csr -out domain.crt -days 365 -CAcreateserial \
    -extfile <(echo "subjectAltName=DNS:$FQDN")
# Print CRT
openssl x509 -text -noout -in domain.crt

Create config files:

# mtls/admission.yaml 
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: ValidatingAdmissionWebhook
  configuration:
    apiVersion: apiserver.config.k8s.io/v1
    kind: WebhookAdmissionConfiguration
    kubeConfigFile: "/etc/mtls/kubeconfig"
- name: MutatingAdmissionWebhook
  configuration:
    apiVersion: apiserver.config.k8s.io/v1
    kind: WebhookAdmissionConfiguration
    kubeConfigFile: "/etc/mtls/kubeconfig"

# mtls/kubeconfig 
apiVersion: v1
kind: Config
users:
- name: 'rancher-webhook.cattle-system.svc'
  user:
    client-certificate: /etc/mtls/domain.crt
    client-key: /etc/mtls/domain.key

Provide configuration and volume to k3s:

k3d cluster create mtls-cluster \
  --k3s-arg '--kube-apiserver-arg=admission-control-config-file=/etc/mtls/admission.yaml@server:*' \
  --volume '/home/username/mtls:/etc/mtls@server:*'

Create configmap and install kubewarden with mTLS support:

kubectl create configmap -n kubewarden mtlscm --from-file=client-ca.crt=mtls/rootCA.crt

# Install controller (helm chart from this PR)
helm install --wait -n kubewarden kubewarden-controller ./charts/kubewarden-controller \
  --set mTLS.enable=true --set mTLS.configMapName=mtlscm \
  --set image.tag=latest --set auditScanner.image.tag=latest

Workaround & Bug:

Controller looks for client-ca.crt in a mount from secret where it does not exist.
If I add it there issue is fixed, but I am not sure it's the place we wanted.

kubectl patch secret -n kubewarden kubewarden-webhook-server-cert --type='merge' -p "{\"data\":{\"client-ca.crt\":\"$(cat mtls/rootCA.crt | base64 -w0)\"}}"
k delete pod -n kubewarden kubewarden-controller-656c6b45f-rc4gx

After this controller starts, but kubewarden-defaults shows another issues:

helm install --wait -n kubewarden kubewarden-defaults ./charts/kubewarden-defaults --set recommendedPolicies.enabled=true --set policyServer.image.tag=latest
Error: INSTALLATION FAILED: 7 errors occurred:
	* Internal error occurred: failed calling webhook "mclusteradmissionpolicy.kb.io": failed to call webhook: Post "https://kubewarden-controller-webhook-service.kubewarden.svc:443/mutate-policies-kubewarden-io-v1-clusteradmissionpolicy?timeout=10s": remote error: tls: certificate required
...
	* Internal error occurred: failed calling webhook "mpolicyserver.kb.io": failed to call webhook: Post "https://kubewarden-controller-webhook-service.kubewarden.svc:443/mutate-policies-kubewarden-io-v1-policyserver?timeout=10s": remote error: tls: certificate required
	* ```

[viccuad]

Currently, I'm getting "error":"failed to read client CA cert: open /tmp/k8s-webhook-server/serving-certs/client-ca.crt: no such file or directory" when enabling mTLS and providing the configmap. I may be doing something incorrectly on my side..

[kravciak]

Currently, I'm getting "error":"failed to read client CA cert: open /tmp/k8s-webhook-server/serving-certs/client-ca.crt: no such file or directory","stacktrace" when enabling mTLS and providing the configmap. I may be doing something incorrectly on my side..

I updated howto with complete steps. I got the same result as you.
@fabriziosestito could you try #638 (comment) - I think there is a bug in controller (see workaround in the comment).

[kravciak]

@fabriziosestito found an issue with the chart (missing mount) and I am fixing an issue with otel tests

[viccuad]

Amended the helm-chart with the correct mounting point for the upcoming controller change.

[viccuad]

The kubewarden-controller:latest image now contains the needed fixes for this PR.

To summarize so far what I'm doing:

Toggle me

Create certs:

#!/usr/bin/env bash
set -aeEuo pipefail

FQDN=mtls.kubewarden.io

# Create CA
openssl req -nodes -batch -x509 -sha256 -days 365 -newkey rsa:2048 -keyout rootCA.key -out rootCA.crt
# Create CSR
openssl req -nodes -batch -newkey rsa:2048 -keyout domain.key -out domain.csr \
  -addext "subjectAltName = DNS:$FQDN"
# Create CRT
openssl x509 -req -CA rootCA.crt -CAkey rootCA.key -in domain.csr -out domain.crt -days 365 -CAcreateserial \
  -extfile <(echo "subjectAltName=DNS:$FQDN")
# Print CRT
openssl x509 -text -noout -in domain.crt

Creates rootCA.{crt,key,srl} and domain.{crt,ket,srl}.

Needed files:

# ./admission.yaml
---
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: ValidatingAdmissionWebhook
    configuration:
      apiVersion: apiserver.config.k8s.io/v1
      kind: WebhookAdmissionConfiguration
      kubeConfigFile: "/etc/mtls/kubeconfig"
  - name: MutatingAdmissionWebhook
    configuration:
      apiVersion: apiserver.config.k8s.io/v1
      kind: WebhookAdmissionConfiguration
      kubeConfigFile: "/etc/mtls/kubeconfig"

# ./kubeconfig
---
apiVersion: v1
kind: Config
users:
  # Config for the controller webhook servive, in the shape of
  # <controller chart name>-webhook-service.<kubewarden ns>.svc:443
  - name: "kubewarden-controller-webhook-service.kubewarden.svc:443"
    user:
      client-certificate: /etc/mtls/domain.crt
      client-key: /etc/mtls/domain.key
  # Config for all policy servers, in the shape of
  # policy-server-<policy-server name>.<policy-server ns>.svc:8443
  - name: "*.kubewarden.svc:8443"
    user:
      client-certificate: /etc/mtls/domain.crt
      client-key: /etc/mtls/domain.key

# ./mtls-configmap.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mtls-configmap
  namespace: kubewarden
data:
  client-ca.crt: |
     <rootCA.crt in PEM format, not base64>

Then, do:

$ k3d cluster create \
  --k3s-arg '--kube-apiserver-arg=admission-control-config-file=/etc/mtls/admission.yaml@server:*' \
  --volume "$(pwd):/etc/mtls@server:*"

Apply the configmap:

$ kubectl apply -f mtls-configmap.yml

Upgrade the helm chart:

$ helm upgrade -i --wait --namespace kubewarden --create-namespace kubewarden-controller \
  ./charts/kubewarden-controller \
  --set auditScanner.cronJob.schedule="*/2 * * * *" --set auditScanner.policyReporter=true \
  --set mTLS.enable=true --set mTLS.configMapName=mtls-configmap \
  --set image.tag=latest

With this, I see the correct logs in policy-server:

INFO policy_server::certs: Loaded client CA certificates client_ca_certs_added=1

Yet creating a policy gives me:

Error from server (InternalError): error when creating "policy-env-var-secrets.yaml": Internal error occurred: failed calling webhook "mclusteradmissionpolicy.kb.io": failed to call webhook: Post "https://kubewarden-controller-webhook-service.kubewarden.svc:443/mutate-policies-kubewarden-io-v1-clusteradmissionpolicy?timeout=10s": remote error: tls: certificate required

[flavio]

@viccuad I think this is failing because the request is sent to https://kubewarden-controller-webhook-service.kubewarden.svc:443, but the kubeconfig has this directive:

- name: "kubewarden-controller-webhook-service.kubewarden.svc.cluster.local:443"

If I were you, I would drop the .cluster.local and try again. I think you can also omit the specify the :443 since it's the default one for https.

[viccuad]

Thanks! I left the :443 for completeness, indeed can be out. Updated the instructions on the other message for completion, rebased changing the order of the volumes in the template.

Policy-server itself works fine.
Audit scanner seems to have an incorrect client cert though. From audit-scanner logs:

"error":"Post \"https://policy-server-default.kubewarden.svc:8443/audit/clusterwide-no-privileged-pod\": remote error: tls: error decrypting message

[viccuad]

Updated tests and one variable assignment on the shared commit, ready for review.
We should possibly merge the e2e test PR, configure it, and rebase this PR.

Current e2e tests check the non-mtls enabled path.

Tested locally with an mTLS cluster. Needs controller:latest. Example of test:

helm upgrade -i --wait --namespace kubewarden --create-namespace \
  kubewarden-controller ./charts/kubewarden-controller \
  --set auditScanner.cronJob.schedule="*/2 * * * *" \
  --set mTLS.enable=true --set mTLS.configMapName=mtls-configmap --set image.tag=latest

Where the mtls-configmap is created with:

kubectl create configmap -n kubewarden mtls-configmap --from-file=client-ca.crt=./rootCA.crt
# rootCA.crt being the root CA used for setting the api-server

Then the audit-scanner should be working.