Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Operator] Grafana Agent does not pick Probe's metric relabeling #6974

Open
MGudin opened this issue Jul 4, 2024 · 0 comments
Open

[Operator] Grafana Agent does not pick Probe's metric relabeling #6974

MGudin opened this issue Jul 4, 2024 · 0 comments

Comments

@MGudin
Copy link

MGudin commented Jul 4, 2024

Hi! I've deployed the operator alongside a blackbox-exporter on an eks cluster, and i'm scraping blackbox targets thru Probe CRD. That works fine.

What's wrong?

Now i'm trying to add custom labels to each probe via the metricRelabelings config block on probe's CRD but it seems that the agent is ignoring it.

Software version

grafana/agent-operator:v0.39.0

Configuration

Example:

apiVersion: monitoring.grafana.com/v1alpha1
kind: GrafanaAgent
metadata:
  name: blackbox
spec:
  disableReporting: true
  disableSupportBundle: false
  enableConfigReadAPI: false
  metrics:
    externalLabels:
      ...
    instanceSelector:
      matchLabels:
        blackbox: ""
    scrapeInterval: 60s
---
apiVersion: monitoring.grafana.com/v1alpha1
kind: MetricsInstance
metadata:
  labels:
    blackbox: ""
  name: blackbox
spec:
  probeSelector:
    matchLabels:
      blackbox: ""
  remoteWrite:
  - headers:
      X-Scope-OrgID: ...
    url: ...
---
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  labels:
    blackbox: ""
  name: myprobe
spec:
  jobName: someprobe
  metricRelabelings:
  - action: replace
    replacement: hello
    targetLabel: mynewlabel
  module: probemodule
  prober:
    path: /probe
    scheme: http
    url: blackbox-exporter:9115
  targets:
    staticConfig:
      static:
      - <target>

What should happen?

Maybe i'm mistaken, but it should create the proper scape config with the relabelConfigs? I checked out the configuration created by the operator and looks like this:

metrics:
    configs:
        - name: <namespace>/blackbox
          remote_write:
            - headers:
                X-Scope-OrgID: ...
              url: ...
          scrape_configs:
            - honor_timestamps: true
              job_name: probe/<namespace>/someprobe
              metrics_path: /probe
              params:
                module:
                    - probemodule
              relabel_configs:
                - source_labels:
                    - job
                  target_label: __tmp_prometheus_job_name
                - replacement: someprobe
                  target_label: job
                - source_labels:
                    - __address__
                  target_label: __param_target
                - source_labels:
                    - __param_target
                  target_label: instance
                - replacement: blackbox-exporter:9115
                  target_label: __address__
              scheme: http
              static_configs:
                - labels:
                    namespace: <namespace>
                  targets:
                    - <target>
    global:
        external_labels:
            __replica__: replica-$(STATEFULSET_ORDINAL_NUMBER)
            ...
        scrape_interval: 60s
    wal_directory: /var/lib/grafana-agent/data
server: {}

I was hoping to see a metric_relabeling_configs block... but it is not showing up. Maybe i'm missing something? Are there any way to achieve what i'm looking for (to add custom labels to metrics via each probe CRD).

Logs

I've checked out operator logs and metrics instance logs and no relevan information there. Just usual reconciles and config reloadings.

Thank's in advance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant