[BETA API] Provide a solution to automatically create TLS secret and inject caBundle

[SaschaSchwarze0]

For the Beta API, #1104 implements a conversion webhook. For a webhook to properly work, we need to create a certificate and let Kubernetes sign it, store it as a Kubernetes Secret, and inject the certificate authority in the caBundle of the conversion webhook section in the CRDs.

We need to investigate an approach that we can still deliver a simple YAML file and the certificate setup magically happens.

Prior art:

• cert-manager does it with its own capabilities here: https://github.com/cert-manager/cert-manager/blob/v1.12.3/deploy/charts/cert-manager/templates/webhook-validating-webhook.yaml#L12
• the Knative controllers create the CAs on their own and update the objects like here for conversion: https://github.com/knative/pkg/blob/release-1.11/webhook/resourcesemantics/conversion/reconciler.go#L103

[adambkaplan]

Cert-manager is attractive since it handles a lot of challenges managing TLS automatically. It has wide adoption amongst the Kubernetes community. The only downside is that it adds a second project dependency to Shipwright - are we okay with that?

[SaschaSchwarze0]

I was also looking at this this evening. Options I see:

1. cert-manager as described by Adam above.
2. knative/pkg is not really an option because then we'd rewrite the whole controller.
3. implementing our own controller logic that uses a csr to request a signed certificate and store that in a secret and the crds. Code for this: https://github.com/Trendyol/k8s-webhook-certificator/blob/main/cmd/root.go. We could do this at startup of the build controller. We would order the certificate when the old one's expiration is less than three months or similar. I would have assumed to find this implemented somewhere by somebody, but have not (yet).
4. logic somewhere near this is explained here: https://www.velotio.com/engineering-blog/managing-tls-certificate-for-kubernetes-admission-webhook. Though, using an initContainer makes no sense once you have high availability (every init container would create it), imo it must be the leader of a controller.
5. We go away from a kubectl apply -f release.yaml and instead have a script that performs the csr handling and applies the manifests.
6. If we would be based on Helm here, we could use a Helm hook, fire up a Job that would take care of the cert handling

tbh, I tend to do (3) because it is not really complicated logic.

[jkhelil]

cert-manager has the advantage to addresse easily both secret creation(providing the certificate and key pair for the webhook) and ca-bundle injection in the crd conversion spec. I like the fact that with this solution, we dont add any code (as far as i understand), so it happens at deployment time, which means systems like openshift who provide mecanism to provide certificate and key pair, to services, we can override easily the use of cert-manager

logic (3) is somehow similar to use knative certificate controller https://github.com/knative/pkg/blob/main/webhook/certificates/controller.go with the advantage to let the certificate controller handle the certificate expiration, rather than implementing our own controller
but still we need crd reconciliation to inject the ca-bundle , so we will need antoher logic to handle that injection like it is done in knative https://github.com/knative/pkg/blob/release-1.11/webhook/resourcesemantics/conversion/reconciler.go#L103

[SaschaSchwarze0]

Discussed in community meeting today:

1. default will be with cert-manager annotations, will generate the CABundle and update the CRD for the four CR's
2. second option is to extend the one-line kubectl apply, and add one more step, where the CABundle is autogenerated with a script and the CR's(4) are patched on the fly.
3. To consider how the above two options will look for the operator.

We look for providing (1) and (2). Meaning those who use cert-manager gain from its convenience. Users who don't (want to) use cert-manager would have to manually manage the certificate, for example using the script that we provide.

fyi @qu1queee @adambkaplan

[SaschaSchwarze0]

I had started to look into (1) last week. Did not completely get it done, but the route is stony anyway. It is not as easy to just put an annotation on the Webhook or CRD. And, CSR support in cert-manager is experimental. https://cert-manager.io/docs/usage/kube-csr/