Unable to scrape metrics for kubevirt-hyperconverged-operator

[nakkoh]

What happened:
A clear and concise description of what the bug is.

Prometheus unable to scrape metrics for kubevirt-hyperconverged-operator.

The cause of this problem appears to be the lack of authorization.

What you expected to happen:
A clear and concise description of what you expected to happen.

How to reproduce it (as minimally and precisely as possible):
Steps to reproduce the behavior.

Deploy KubeVirt HyperConverged Cluster Operator on OKD.

Additional context:
Add any other context about the problem here.

Environment:

• KubeVirt version (use virtctl version): v1.4.0
• Kubernetes version (use kubectl version): v1.31.6
• VM or VMI specifications: N/A
• Cloud provider or hardware configuration:
  • controle plane nodes: OpenStack
  • compute nodes: Baremetal
• OS (e.g. from /etc/os-release): CentOS Stream CoreOS 418.9.202503040632-0
• Kernel (e.g. uname -a): 5.14.0-570.el9.x86_64
• Install tools: N/A
• Others:
  • OKD 4.18.0-okd-scos.3

[orenc1]

Hi @machadovilaca , could you please check?
I suspect it's related to #3303

[machadovilaca]

Hello @nakkoh,

can you share the hco-operator pod logs and the config of the kubevirt-hyperconverged-operator-metrics ServiceMonitor?

[nakkoh]

Thank you @machadovilaca

Please refer to the following attachment regarding hco-operator logs.
hco_operator.log

And the kubevirt-hyperconverged-operator-metrics ServiceMonitor is defined as follows.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  creationTimestamp: "2025-02-28T06:31:10Z"
  generation: 16
  labels:
    app: kubevirt-hyperconverged
    app.kubernetes.io/component: monitoring
    app.kubernetes.io/managed-by: hco-operator
    app.kubernetes.io/part-of: hyperconverged-cluster
    app.kubernetes.io/version: 1.14.0
  name: kubevirt-hyperconverged-operator-metrics
  namespace: kubevirt-hyperconverged
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: false
    controller: false
    kind: Deployment
    name: hco-operator
    uid: 19e2af5c-076d-4bbe-9450-f0d978e830be
  resourceVersion: "19149989"
  uid: b6df0cc7-c4e4-4c30-8039-8b22e0a89c12
spec:
  endpoints:
  - authorization:
      credentials:
        key: token
        name: hco-bearer-auth
    port: http-metrics
  namespaceSelector: {}
  selector:
    matchLabels:
      app: kubevirt-hyperconverged                                                                        
      app.kubernetes.io/component: monitoring
      app.kubernetes.io/managed-by: hco-operator
      app.kubernetes.io/part-of: hyperconverged-cluster
      app.kubernetes.io/version: 1.14.0