cc @zeritti

Thanks, @TheRealNoob, for raising this issue. This is indeed a known problem. Once a mutating/validation webhook configuration is present, the API server will contact the admission webhook just after a PrometheusRule has been created. At an install time, this will fail (assuming a custom CA is being used) as the patch job has not run as yet and the configuration has not been patched with the CA bundle.

I used to work around this problem with two steps when installing - 1. install a release with the PrometheusRule disabled and 2. upgrade the release with the rule enabled. I always preferred using cert-manager wherever possible, though.

Using hook weight would be elegant. In this case, though, the patch job has to patch an existing resource - if the job runs as a pre-install hook at the install time, the resource, i.e. a validating/mutating webhook configuration is not present and the job is going to fail.

Interesting problem....I spent some time thinking about the different approaches. The cleanest solution I came up with was to include the validating/mutating webhookconfiguration objects in the presync. And it turns out someone's already added support for just that in kube-prometheus-stack.

It's possible that this could cause errors if someone is applying prometheusRule/Alertmanager objects in another parallel release, but I'm chalking this up as an acceptable risk to get a clean release.

Thoughts?

So I updated the MR with the proposed changes above. I tested it out on minikube and I found that it does work, but a new problem arose. When helm gets to applying the prometheusRule objects, apiserver makes the call to the operator, and the request times out because the operator is still in startup.

Not sure how to address that issue. Helm doesn't have any concept of forced sleep unless you go crazy with a Job that runs a sleep command. Even so, I still consider the above change an improvement. At least now when we re-run the release with the same values file it works.

We use ArgoCD at my dayjob, not sure how familiar you are with it. If Argo does a release and it fails, it tries it again 4 more times before marking it as failed. I figure at least in this setup it will now succeed one of those times.

Come to think of it, the chart probably has the same issue in the certManager approach

So I updated the MR with the proposed changes above. I tested it out on minikube and I found that it does work, but a new problem arose. When helm gets to applying the prometheusRule objects, apiserver makes the call to the operator, and the request times out because the operator is still in startup.

Yes, this is what happens, indeed. It will be ok in the end if the install gets retried.

The thing with making the configurations into hooks is that as hooks they get deleted, i.e. there will be a moment on upgrades with the configurations being absent. I also think that the configurations won't be deleted with a deletion of the release as they are not part of the release. These points should also be considered.

Sorry, I'm struggling to follow. Are you say that the ValidatingWebhookConfiguration and MutatingWebhookConfiguration objects get deleted and recreated during a chart upgrade? Why would that be? I can't even image there being changes to those objects

Perhaps you're referring to the "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded annotation? I did not include this annotations on the *WebhookConfiguration objects.

Perhaps you're referring to the "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded annotation? I did not include this annotations on the *WebhookConfiguration objects.

Yes, this is what I meant. If you do not set it, the default applies (Ref.):

If no hook deletion policy annotation is specified, the before-hook-creation behavior applies by default.

Ah I see, I didn't know that. I assumed it defaulted to blank. So we need to define it as blank then. I'll give this a try and make sure this works. Assuming it does, you're onboard with this approach?