prometheus: Added basic authorization

[vlasov-y]

Hi!

This PR adds support of basic authorization for prometheus plugin. Nevertheless, there is an issue on the backend side. I have decided to show changes and describe the issue in the video below.

Video review and description of the problem
https://github.com/user-attachments/assets/ee2f14ae-117f-47ce-be2b-bfa1472357b4

[joaquimrocha]

@yolossn It'd great to include this in our release next week. Please let me know if you can review or if someone else should.

[vlasov-y]

@yolossn It'd great to include this in our release next week. Please let me know if you can review or if someone else should.

It has an issue that I have mentioned in the video. Seems that backend does not pass authorization header to the prometheus.

[yolossn]

@vlasov-y I looked into the auth headers not being forwarded issue that you mentioned. Looks like the kubernetes api-server is assuming the authorization header is for itself and not forwarding it to the proxied service or pod. Since the Authorization header is being removed by the kubernetes api-server we should look for an upstream fix.
cc: @joaquimrocha

Sharing the scripts that I used to debug this for your reference

Create a http echo service.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpecho
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpecho
  template:
    metadata:
      labels:
        app: httpecho
    spec:
      containers:
      - name: httpecho
        image: mendhak/http-https-echo:latest
        ports:
        - containerPort: 8443
---
apiVersion: v1
kind: Service
metadata:
  name: httpecho
spec:
  ports:
  - port: 80
    targetPort: 8443
  selector:
    app: httpecho

Once the echo service is deployed create a proxy to api server by running

kubectl proxy

curl -H "Authorization: Basic $(echo -n 'username:password' | base64)" \
  http://localhost:8001/api/v1/namespaces/default/services/httpecho:80/proxy/api/v1/query_range?start=1737014987&end=1737101387&step=300&query=sum%28rate%28container_cpu_usage_seconds_total%7Bcontainer%21%3D%27%27%2Cnamespace%3D%27default%27%2Cpod%3D%27httpecho-56b9f47ccb-qss6m%27%7D%5B1m%5D%29%29+by+%28pod%2Cnamespace%29

curl -H "X-Forwarded-Authorization: Basic $(echo -n 'username:password' | base64)" \
  http://localhost:8001/api/v1/namespaces/default/services/httpecho:80/proxy/api/v1/query_range?start=1737014987&end=1737101387&step=300&query=sum%28rate%28container_cpu_usage_seconds_total%7Bcontainer%21%3D%27%27%2Cnamespace%3D%27default%27%2Cpod%3D%27httpecho-56b9f47ccb-qss6m%27%7D%5B1m%5D%29%29+by+%28pod%2Cnamespace%29

[vlasov-y]

@yolossn thank you for your investigation. Wdyt we have to do with this PR while problem is not fixed in k8s API? By the way, have you created an issue in k8s repo for that already?

[yolossn]

@yolossn thank you for your investigation. Wdyt we have to do with this PR while problem is not fixed in k8s API? By the way, have you created an issue in k8s repo for that already?

I am looking for alternate ways to solve this. I haven't created any issues yet. For now I am looking into the prometheus codebase to check if it accepts any other header(like X-Forwarded-Authorization) other than Authorization. Ill keep you updated.

[vlasov-y]

@yolossn thank you for your investigation. Wdyt we have to do with this PR while problem is not fixed in k8s API? By the way, have you created an issue in k8s repo for that already?

I am looking for alternate ways to solve this. I haven't created any issues yet. For now I am looking into the prometheus codebase to check if it accepts any other header(like X-Forwarded-Authorization) other than Authorization. Ill keep you updated.

Thanks! I have skimmed over issues, we are not first kubernetes/kubernetes#38775 (comment)
I will read couple more issues, maybe they have built the solution already

[vlasov-y]

@yolossn It looks like we've hit a dead end. I've done a few more checks:

1. When querying Caddy directly (with and without authorization), CORS headers are returned correctly. However, when going through the kube-api, they are missing.
2. We could add --requestheader-extra-headers-prefix=X-Remote-Extra- to the kube-apiserver extra args, which might help with the Authorization header issue but that’s overkill for this case.

Not sure if there's a better workaround. Any thoughts?

[yolossn]

@yolossn It looks like we've hit a dead end. I've done a few more checks:

Yes, seems like there isn't any easy workaround for this. Ill search if there are any other workarounds without having to make major changes to the plugin itself. If nothing works maybe we should consider creating a port-forward to the prometheus service and then use it for querying the metrics but this solution will make the plugin unusable for in-cluster deployments of headlamp.

[joaquimrocha]

Moving to Draft since it needs some further investigation.