Dedicated webserver for health check endpoint

[tejaswiniVadlamudi]

Prometheus with direct TLS support (by using --web.config.file flag) expects strict client certificates for all HTTP endpoint consumers.
When deployed on k8s, k8s (kubelet) doesn't provide a client certificate when it probes readiness and liveness endpoints.
One needs to disable health check probes in the deployment manifests in order to deploy Prometheus with inbuilt TLS support on k8s.

See k8s documentation here (https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes). It says "'if the scheme field is set to HTTPS, the kubelet sends an HTTPS request skipping the certificate verification"

Can we move /-/ready and /-/healthy endpoints to a separate web server?

[roidelapluie]

It would be acceptable to me to be able to somehow allow /-/ready and /-/healthy to not check for certificate.

[tejaswiniVadlamudi]

@roidelapluie : Do you accept code contributions for this task? If yes, could you hint me at the next step?

[roidelapluie]

On 08 Jun 15:42, tejaswiniVadlamudi wrote: @roidelapluie : Do you accept code contributions for this task? If yes, could you hint me at the next step? -- Reply to this email directly or view it on GitHub: #111 You are receiving this because you were mentioned. Message ID: ***@***.***>

The next step would be a pull request on the exporter-toolkit repository with an option: "allow_anonymous_probes: true" to enable readiness and liveness probes without authentication.

…

-- Julien Pivotto O11y - https://o11y.eu/

[roidelapluie]

We are working on this #106

[twiden]

I think it would be really good to be able to exclude /-/healthy and /-/ready from basic auth and possibly TLS verification as well. My usecase is that I run Pushgateway on ECS and basic auth prevents the Application Load Balancer to perform meaningful health checks. I also only push metrics to the gateway on the local subnet so in my case a good solution would be to bypass basic auth for health, readiness and metrics push and maybe add ip whitelisting based on cidr for those endpoints instead.