Grafana Agent not sending metrics with operator method

[luvpreetsingh]

Hi,

I have been using grafana-agent in my staging environment using this method and it was working fine.

But then I wanted to get stats from my nodejs apps which have prometheus middleware installed in them so I needed to add some ServiceMonitors. For that I deleted this current installation of grafana agent and installed the grafana-agent operator using this method and then did the whole setup using this method. I installed everything in monitoring namespace so helm install had -n monitoring flag with it.

Here is my config,

FOR CRDs

apiVersion: monitoring.grafana.com/v1alpha1
kind: GrafanaAgent
metadata:
  name: grafana-agent
  namespace: monitoring
  labels:
    app: grafana-agent
spec:
  image: grafana/agent:v0.26.1
  logLevel: info
  serviceAccountName: grafana-agent
  metrics:
    instanceSelector:
      matchLabels:
        agent: grafana-agent-metrics
    # I want clustername as my label with all metrics
    externalLabels:
      clustername: greenlock-staging

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: grafana-agent
  namespace: monitoring

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: grafana-agent
rules:
  - apiGroups:
    - ""
    resources:
    - nodes
    - nodes/proxy
    - nodes/metrics
    - services
    - endpoints
    - pods
    - events
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - networking.k8s.io
    resources:
    - ingresses
    verbs:
    - get
    - list
    - watch
  - nonResourceURLs:
    - /metrics
    - /metrics/cadvisor
    verbs:
    - get

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: grafana-agent
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: grafana-agent
subjects:
- kind: ServiceAccount
  name: grafana-agent
  namespace: monitoring

For MetricInstance

apiVersion: monitoring.grafana.com/v1alpha1
kind: MetricsInstance
metadata:
  name: primary
  namespace: monitoring
  labels:
    agent: grafana-agent-metrics
spec:
  remoteWrite:
  - url: http://prometheus-server-headless.monitoring.svc.clusterset.local:9090/api/v1/write
  # monitor all namespaces
  serviceMonitorNamespaceSelector: {}
  podMonitorNamespaceSelector: {}
  probeNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      instance: primary
  podMonitorSelector:
    matchLabels:
      instance: primary
  probeSelector:
    matchLabels:
      instance: primary

For ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    instance: primary
  name: kubelet-monitor
  namespace: default
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    interval: 60s
    metricRelabelings:
    - action: keep
      regex: kubelet_cgroup_manager_duration_seconds_count|go_goroutines|kubelet_pod_start_duration_seconds_count|kubelet_runtime_operations_total|kubelet_pleg_relist_duration_seconds_bucket|volume_manager_total_volumes|kubelet_volume_stats_capacity_bytes|container_cpu_usage_seconds_total|container_network_transmit_bytes_total|kubelet_runtime_operations_errors_total|container_network_receive_bytes_total|container_memory_swap|container_network_receive_packets_total|container_cpu_cfs_periods_total|container_cpu_cfs_throttled_periods_total|kubelet_running_pod_count|node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate|container_memory_working_set_bytes|storage_operation_errors_total|kubelet_pleg_relist_duration_seconds_count|kubelet_running_pods|rest_client_request_duration_seconds_bucket|process_resident_memory_bytes|storage_operation_duration_seconds_count|kubelet_running_containers|kubelet_runtime_operations_duration_seconds_bucket|kubelet_node_config_error|kubelet_cgroup_manager_duration_seconds_bucket|kubelet_running_container_count|kubelet_volume_stats_available_bytes|kubelet_volume_stats_inodes|container_memory_rss|kubelet_pod_worker_duration_seconds_count|kubelet_node_name|kubelet_pleg_relist_interval_seconds_bucket|container_network_receive_packets_dropped_total|kubelet_pod_worker_duration_seconds_bucket|container_start_time_seconds|container_network_transmit_packets_dropped_total|process_cpu_seconds_total|storage_operation_duration_seconds_bucket|container_memory_cache|container_network_transmit_packets_total|kubelet_volume_stats_inodes_used|up|rest_client_requests_total
      sourceLabels:
      - __name__
    - action: replace
      targetLabel: job
      replacement: integrations/kubernetes/kubelet
    port: https-metrics
    relabelings:
    - sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app.kubernetes.io/name: kubelet

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    instance: primary
  name: cadvisor-monitor
  namespace: default
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    honorLabels: true
    honorTimestamps: false
    interval: 60s
    metricRelabelings:
    - action: keep
      regex: kubelet_cgroup_manager_duration_seconds_count|go_goroutines|kubelet_pod_start_duration_seconds_count|kubelet_runtime_operations_total|kubelet_pleg_relist_duration_seconds_bucket|volume_manager_total_volumes|kubelet_volume_stats_capacity_bytes|container_cpu_usage_seconds_total|container_network_transmit_bytes_total|kubelet_runtime_operations_errors_total|container_network_receive_bytes_total|container_memory_swap|container_network_receive_packets_total|container_cpu_cfs_periods_total|container_cpu_cfs_throttled_periods_total|kubelet_running_pod_count|node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate|container_memory_working_set_bytes|storage_operation_errors_total|kubelet_pleg_relist_duration_seconds_count|kubelet_running_pods|rest_client_request_duration_seconds_bucket|process_resident_memory_bytes|storage_operation_duration_seconds_count|kubelet_running_containers|kubelet_runtime_operations_duration_seconds_bucket|kubelet_node_config_error|kubelet_cgroup_manager_duration_seconds_bucket|kubelet_running_container_count|kubelet_volume_stats_available_bytes|kubelet_volume_stats_inodes|container_memory_rss|kubelet_pod_worker_duration_seconds_count|kubelet_node_name|kubelet_pleg_relist_interval_seconds_bucket|container_network_receive_packets_dropped_total|kubelet_pod_worker_duration_seconds_bucket|container_start_time_seconds|container_network_transmit_packets_dropped_total|process_cpu_seconds_total|storage_operation_duration_seconds_bucket|container_memory_cache|container_network_transmit_packets_total|kubelet_volume_stats_inodes_used|up|rest_client_requests_total
      sourceLabels:
      - __name__
    - action: replace
      targetLabel: job
      replacement: integrations/kubernetes/cadvisor
    path: /metrics/cadvisor
    port: https-metrics
    relabelings:
    - sourceLabels:
      - __metrics_path__
      targetLabel: metrics_path
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  namespaceSelector:
    matchNames:
    - default
  selector:
    matchLabels:
      app.kubernetes.io/name: kubelet

When I installed using new method, data stopped appearing in prometheus. Here is a screenshot of the same,

The screenshot is of the metric kubelet_http_requests_total. I tried with other metrics too and it was the same result. Here was the full query,

kubelet_http_requests_total{clustername="greenlock-staging"}

In case you need my old configmap with the old setup, here it is

kind: ConfigMap
metadata:
  name: grafana-agent
  namespace: monitoring
apiVersion: v1
data:
  agent.yaml: |
    metrics:
      wal_directory: /var/lib/agent/wal
      global:
        scrape_interval: 15s
        external_labels:
          clustername: greenlock-staging
      configs:
      - name: integrations
        remote_write:
          - url: http://prometheus-server-headless.monitoring.svc.clusterset.local:9090/api/v1/write
        scrape_configs:
        - job_name: integrations/kubernetes/cadvisor
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          kubernetes_sd_configs:
            - role: node
          relabel_configs:
            - replacement: kubernetes.default.svc:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
              source_labels:
                - __meta_kubernetes_node_name
              target_label: __metrics_path__
          scheme: https
          tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
              server_name: kubernetes
        - job_name: integrations/kubernetes/kubelet
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          kubernetes_sd_configs:
            - role: node
          relabel_configs:
            - replacement: kubernetes.default.svc:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/$1/proxy/metrics
              source_labels:
                - __meta_kubernetes_node_name
              target_label: __metrics_path__
          scheme: https
          tls_config:
              ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              insecure_skip_verify: false
              server_name: kubernetes
        - job_name: integrations/kubernetes/pod-metrics
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - action: drop
              regex: kube-state-metrics
              source_labels:
                - __meta_kubernetes_pod_label_app_kubernetes_io_name
            - action: labelmap
              regex: __meta_kubernetes_pod_label_(.+)
            - source_labels: [__meta_kubernetes_namespace]
              action: replace
              target_label: namespace
            - source_labels: [__meta_kubernetes_pod_name]
              action: replace
              target_label: pod
            - source_labels: ['__meta_kubernetes_namespace', '__meta_kubernetes_pod_label_name']
              action: 'replace'
              separator: '/'
              target_label: 'job'
              replacement: '$1'
            - source_labels: ['__meta_kubernetes_pod_container_name']
              action: 'replace'
              target_label: 'container'
        - job_name: integrations/kubernetes/kube-state-metrics
          kubernetes_sd_configs:
            - role: endpoints

Here are the logs from grafana-agent pod

ts=2022-08-11T09:55:46.540939919Z caller=server.go:191 level=info msg="server listening on addresses" http=[::]:8080 grpc=127.0.0.1:12346 http_tls_enabled=false grpc_tls_enabled=false
ts=2022-08-11T09:55:46.541510689Z caller=node.go:85 level=info agent=prometheus component=cluster msg="applying config"
ts=2022-08-11T09:55:46.541603349Z caller=remote.go:180 level=info agent=prometheus component=cluster msg="not watching the KV, none set"
ts=2022-08-11T09:55:46.543323349Z caller=integrations.go:138 level=warn msg="integrations-next is enabled. integrations-next is subject to change"
ts=2022-08-11T09:55:46Z level=info caller=traces/traces.go:143 msg="Traces Logger Initialized" component=traces
ts=2022-08-11T09:55:46.544983388Z caller=reporter.go:107 level=info msg="running usage stats reporter"
ts=2022-08-11T09:55:46.548815198Z caller=wal.go:197 level=info agent=prometheus instance=e7ad92215fc7fc6f26786aff69d6f266 msg="replaying WAL, this may take a while" dir=/var/lib/grafana-agent/data/e7ad92215fc7fc6f26786aff69d6f266/wal
ts=2022-08-11T09:55:46.549020558Z caller=wal.go:244 level=info agent=prometheus instance=e7ad92215fc7fc6f26786aff69d6f266 msg="WAL segment loaded" segment=0 maxSegment=0
ts=2022-08-11T09:55:46.549742258Z caller=dedupe.go:112 agent=prometheus instance=e7ad92215fc7fc6f26786aff69d6f266 component=remote level=info remote_name=e7ad92-9efa8c url=http://prometheus-server-headless.monitoring.svc.clusterset.local:9090/api/v1/write msg="Starting WAL watcher" queue=e7ad92-9efa8c
ts=2022-08-11T09:55:46.549772668Z caller=dedupe.go:112 agent=prometheus instance=e7ad92215fc7fc6f26786aff69d6f266 component=remote level=info remote_name=e7ad92-9efa8c url=http://prometheus-server-headless.monitoring.svc.clusterset.local:9090/api/v1/write msg="Starting scraped metadata watcher"
ts=2022-08-11T09:55:46.550010888Z caller=dedupe.go:112 agent=prometheus instance=e7ad92215fc7fc6f26786aff69d6f266 component=remote level=info remote_name=e7ad92-9efa8c url=http://prometheus-server-headless.monitoring.svc.clusterset.local:9090/api/v1/write msg="Replaying WAL" queue=e7ad92-9efa8c
ts=2022-08-11T09:55:46.838394696Z caller=entrypoint.go:249 level=info msg="reload of config file requested"
ts=2022-08-11T09:59:58.685807472Z caller=entrypoint.go:249 level=info msg="reload of config file requested"
ts=2022-08-11T09:59:58.697828981Z caller=kubernetes.go:313 level=info agent=prometheus instance=e7ad92215fc7fc6f26786aff69d6f266 component="discovery manager" discovery=kubernetes msg="Using pod service account via in-cluster config"
ts=2022-08-11T09:59:58.698252061Z caller=manager.go:198 level=info agent=prometheus msg="dynamically updated instance" instance=e7ad92215fc7fc6f26786aff69d6f266
ts=2022-08-11T10:00:03.842299264Z caller=dedupe.go:112 agent=prometheus instance=e7ad92215fc7fc6f26786aff69d6f266 component=remote level=info remote_name=e7ad92-9efa8c url=http://prometheus-server-headless.monitoring.svc.clusterset.local:9090/api/v1/write msg="Done replaying WAL" duration=4m17.292358316s
ts=2022-08-11T10:55:46.562037627Z caller=wal.go:407 level=info agent=prometheus instance=e7ad92215fc7fc6f26786aff69d6f266 msg="series GC completed" duration=11.729489ms

Here are prometheus logs for the last 2 hours. I made the change about an hour ago.

$ kubectl logs prometheus-server-0 -n monitoring -c prometheus-server -f --since=2h
ts=2022-08-11T11:00:13.488Z caller=compact.go:519 level=info component=tsdb msg="write block" mint=1660204800003 maxt=1660212000000 ulid=01GA68Z91BPJWARG5QCVF99JPW duration=13.317380922s
ts=2022-08-11T11:00:13.506Z caller=db.go:1292 level=info component=tsdb msg="Deleting obsolete block" block=01GA3FPTQBF75DCKPF9MAAJHXR
ts=2022-08-11T11:00:15.048Z caller=head.go:840 level=info component=tsdb msg="Head GC completed" duration=1.542031233s
ts=2022-08-11T11:00:15.103Z caller=checkpoint.go:97 level=info component=tsdb msg="Creating checkpoint" from_segment=531 to_segment=539 mint=1660212000000
ts=2022-08-11T11:00:23.152Z caller=head.go:1009 level=info component=tsdb msg="WAL checkpoint complete" first=531 last=539 duration=8.04961295s

Is there anything I am doing wrong? Anything I'm missing perhaps? Thanks

[luvpreetsingh]

Please don't bother with this. There was prometheus-kube-stack helm chart deployed in another namespace. I deleted that and it started working. Thanks :)

[mykolaveremeichyk]

Hi @luvpreetsingh.
I'm experiencing the same problem you described here.
Did you have a chance to find out why the grafana agent operator method didn't work with prometheus-kube-stack helm chart?
I don't have this chart deployed to my cluster, but grafana agent doesn't send metrics to the destination.

Thanks in advance.