Metrics not showing up from cert-manager and traefik

[Drabuna]

Hi!

I followed the quick start guide to setup Grafana Cloud Agent on a Kubernetes cluster (running on Digital Ocean).
First, I generated YAML files using the deployment shell scripts, and then I applied them.
I can see metrics and logs in my Grafana UI.

I also have traefik and cert-manager running in my cluster.
I can see logs from both traefik and cert-manager.
When exploring up metrics in the UI, I can see instances of traefik, but not cert-manager.
When I port-forward, I can see /metrics displaying data for both of them.

Here's how I configured scraping in the agent:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: grafana-agent
  namespace: infra
---
apiVersion: v1
data:
  agent.yml: |
    prometheus:
        configs:
          - name: integrations
            scrape_configs:
              - job_name: cert-integration
                static_configs:
                  - targets: ['cert-manager:9402']
              - job_name: traefik-integration
                static_configs:
                  - target: ['traefik:9090']
          - host_filter: true
            name: agent
#... the rest of standard template      

Agent, traefik and cert-manager are in the same namespace (if that matters).
After applying, I did rolling restart of both deployment/grafana-agent-deployment and daemonset/grafana-agent

Metrics won't show up :(

Help would be greatly appreciated :)

[rfratto]

Hey! Can you port-forward to the HTTP server used by the Agent and query for the targets API?

[rfratto]

Gotcha. Can you share the output of the targets API and re-confirm that if you exec into the deployment pod and curling http://cert-manager:9402/metrics gives you a list of metrics?

[Drabuna]

Sure!

Here's the ouput:

{
   "status":"success",
   "data":[
      {
         "instance":"bb0ef566ce395d7993b5f7d1c93fd3c9",
         "target_group":"default/kubernetes",
         "endpoint":"https://<ip>:443/metrics",
         "state":"up",
         "labels":{
            "instance":"<ip>:443",
            "job":"default/kubernetes"
         },
         "last_scrape":"2021-02-02T17:01:41.725441511Z",
         "scrape_duration_ms":118,
         "scrape_error":""
      },
      {
         "instance":"fb740871c312f021ace94687f9fe32e8",
         "target_group":"cert-integration",
         "endpoint":"http://cert-manager:9402/metrics",
         "state":"up",
         "labels":{
            "instance":"cert-manager:9402",
            "job":"cert-integration"
         },
         "last_scrape":"2021-02-02T17:01:40.576891165Z",
         "scrape_duration_ms":5,
         "scrape_error":""
      },
      {
         "instance":"fb740871c312f021ace94687f9fe32e8",
         "target_group":"traefik-integration",
         "endpoint":"http://traefik-metrics:9090/metrics",
         "state":"up",
         "labels":{
            "instance":"traefik-metrics:9090",
            "job":"traefik-integration"
         },
         "last_scrape":"2021-02-02T17:01:42.743934046Z",
         "scrape_duration_ms":4,
         "scrape_error":""
      }
   ]
}

And yes, curling from pod returns data for both traefik and cert-manager.

[rfratto]

Ahh, sorry, I missed the obvious here. Each instance config needs its own remote_write section, can you add it to the integrations config? You can just copy it from the other one.

(We're planning on making a global remote_write config block, but haven't done it yet)

[Drabuna]

Yep, working now 👍
Thanks so much for your help :)

P.S.: I guess it can be useful to improve the documentation a bit since for a complete Grafana stack newbie this was pretty difficult to figure out :)
P.P.S.: Also, charts with "minimal" metrics would make sense, since it can have huge financial consequences for startups and solo-developers.

[rfratto]

You're completely right, and better documentation and better defaults are both on our roadmap for this quarter :) Thanks for the feedback!