ServiceMonitor: Invalid Configuration

[rbaumgar]

On my OpenShift 4.16 with OpenShift Lightspeed Operator 0.1.2

ServiceMonitor lightspeed-app-server-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits
ServiceMonitor lightspeed-operator-controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via tls config which Prometheus specification prohibits

$ oc get event -n openshift-lightspeed 
LAST SEEN   TYPE      REASON                 OBJECT                                                                  MESSAGE
29m         Warning   InvalidConfiguration   servicemonitor/lightspeed-app-server-monitor                            ServiceMonitor lightspeed-app-server-monitor was rejected due to invalid configuration: it accesses file system via bearer token file which Prometheus specification prohibits
29m         Warning   InvalidConfiguration   servicemonitor/lightspeed-operator-controller-manager-metrics-monitor   ServiceMonitor lightspeed-operator-controller-manager-metrics-monitor was rejected due to invalid configuration: it accesses file system via tls config which Prometheus specification prohibits

$ oc get servicemonitors.monitoring.coreos.com -n openshift-lightspeed -o yaml|oc neat
- apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    labels:
      app.kubernetes.io/component: metrics
      app.kubernetes.io/managed-by: lightspeed-operator
      app.kubernetes.io/name: lightspeed-service-api
      app.kubernetes.io/part-of: openshift-lightspeed
      monitoring.openshift.io/collection-profile: full
    name: lightspeed-app-server-monitor
    namespace: openshift-lightspeed
  spec:
    endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      interval: 30s
      path: /metrics
      port: https
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
        certFile: /etc/prometheus/secrets/metrics-client-certs/tls.crt
        keyFile: /etc/prometheus/secrets/metrics-client-certs/tls.key
        serverName: lightspeed-app-server.openshift-lightspeed.svc
    jobLabel: app.kubernetes.io/name
    selector:
      matchLabels:
        app.kubernetes.io/component: application-server
        app.kubernetes.io/managed-by: lightspeed-operator
        app.kubernetes.io/name: lightspeed-service-api
        app.kubernetes.io/part-of: openshift-lightspeed
- apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    labels:
      app.kubernetes.io/component: metrics
      app.kubernetes.io/created-by: lightspeed-operator
      app.kubernetes.io/instance: controller-manager-metrics-monitor
      app.kubernetes.io/managed-by: kustomize
      app.kubernetes.io/name: servicemonitor
      app.kubernetes.io/part-of: lightspeed-operator
      control-plane: controller-manager
      olm.managed: "true"
    name: lightspeed-operator-controller-manager-metrics-monitor
    namespace: openshift-lightspeed
  spec:
    endpoints:
    - path: /metrics
      port: metrics
      scheme: https
      tlsConfig:
        caFile: /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt
        certFile: /etc/prometheus/secrets/metrics-client-certs/tls.crt
        insecureSkipVerify: false
        keyFile: /etc/prometheus/secrets/metrics-client-certs/tls.key
        serverName: lightspeed-operator-controller-manager-service.openshift-lightspeed.svc
    selector:
      matchLabels:
        control-plane: controller-manager

[raptorsun]

Thank you for raising the issue.
Could you please share the OLSConfig CR that produces this problem?
I cannot reproduce the problem with Openshift 4.16.

Meanwhile, please try upgrade to version 0.1.3 to see whether this issue persists.

[xiormeesh]

I found this thread because I was fixing the same issue in another project on 4.16, it's caused by .spec.endpoints[] bearerTokenFile being deprecated in 4.16, however it should give a warning for now and not block installation, maybe there is something forcing fails on deprecations on OP's cluster.

[rbaumgar]

In the meantime, the cluster upgraded to version 0.1.3.
Alert didn't go away.
And yes, it is not blocking, it is found by an alert, PrometheusOperatorRejectedResources

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[rbaumgar]

still open, even with lightspeed 0.2.1

level=warn ts=2025-01-02T07:27:03.376846094Z caller=resource_selector.go:126 component=prometheus-controller msg="skipping servicemonitor" error="it accesses file system via bearer token file which Prometheus specification prohibits" servicemonitor=openshift-lightspeed/lightspeed-app-server-monitor namespace=openshift-user-workload-monitoring prometheus=user-workload

[rbaumgar]

/remove-lifecycle stale

[raptorsun]

Happy new year! Thank you for reminding us of this issue!
A Jira ticket is written to track this issue https://issues.redhat.com/browse/OLS-1322

[raptorsun]

/remove-lifecycle rotten

[raptorsun]

Hello @rbaumgar , in the cluster having this issue

1. does the namespace openshfit-lightspeed have the label openshift.io/cluster-monitoring: "true"?
2. does the cluster have user workload monitoring stack setup?

Here is a related runbook on the alert PrometheusOperatorRejectedResources : https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/PrometheusOperatorRejectedResources.md#servicemonitor-and-podmonitor

[raptorsun]

Could you please share the content of the configmap cluster-monitoring-config in the namespace openshift-monitoring ?

[rbaumgar]

@raptorsun

1. when the label is set the error is gone.
2. yes

cluster-monitoring-config

config.yaml:
----
enableUserWorkload: true
# Parameters for Platform prometheus
prometheusK8s:
  # retention: 15d
  # retentionSize: 90GB
  # volumeClaimTemplate:
  #   spec:
  #     resources:
  #       requests:
  #         storage: 100Gi

[raptorsun]

Thank you for the timely return.

Adding the label openshift.io/cluster-monitoring: "true" will make the in-cluster Prometheus (Platform prometheus) scrape metrics from the namespace openshift-lightspeed. In that namespace there is no such restriction to access token file via filesystem.

As user-workload-monitoring is activated, the alert should come from the UMW Prometheus that uses different settings from the in-cluster one.

To gather more details from this issue, could you check whether the ServiceMonitors in the namespace openshift-lightspeed have the label openshift.io/user-monitoring: "false"?

[rbaumgar]

controller yes

$ oc get servicemonitors.monitoring.coreos.com -n openshift-lightspeed --show-labels 
NAME                                                     AGE    LABELS
lightspeed-app-server-monitor                            116d   app.kubernetes.io/component=metrics,app.kubernetes.io/managed-by=lightspeed-operator,app.kubernetes.io/name=lightspeed-service-api,app.kubernetes.io/part-of=openshift-lightspeed,monitoring.openshift.io/collection-profile=full
lightspeed-operator-controller-manager-metrics-monitor   116d   app.kubernetes.io/component=metrics,app.kubernetes.io/created-by=lightspeed-operator,app.kubernetes.io/instance=controller-manager-metrics-monitor,app.kubernetes.io/managed-by=kustomize,app.kubernetes.io/name=servicemonitor,app.kubernetes.io/part-of=lightspeed-operator,control-plane=controller-manager,olm.managed=true,openshift.io/user-monitoring=false