Monitoring infrastructure next level with Mimir 🚀

[mcbenjemaa]

Context

In order to solve limitations of our current monitoring setup, we need to try new tools that will basically help us for long term storage metrics, and HA, as well as Global view Querying...

One of the best tools in the market is Grafana Mimir.
We need to Try it, and find the best way to integrate it in our setup.

User Story

As a Giant Swarm engineer, I want to be able to access a history of metrics for components managing our platform in order to be to both investigate ongoing operational issues and provide details for incident reports.

Ideally a month of metrics history should be available.

Tasks

Tasks

Beta Give feedback

1. Create S3 bucket manually for Mimir #2263
   team/atlas topic/monitoring +2
   
2. Package Mimir into an app #2264
   team/atlas topic/monitoring +2
   
3. Deploy Mimir #2265
   team/atlas topic/monitoring +2
   
4. Ingesting metrics into Mimir #2266
   team/atlas topic/monitoring +2
   
5. Use Mimir as Grafana datasource #2267
   team/atlas topic/monitoring +2
   
6. Build dashboard for Prometheus cost #2430
   component/prometheus team/atlas topic/monitoring +3
   
7. Setup prometheus-rules in Mimir #2521
   team/atlas team/phoenix topic/monitoring +3
   
8. https://github.com/giantswarm/giantswarm/issues/26707
9. Mimir cost dashboard dashboards#305
   +0
10. Connect Mimir metrics with Grafana cloud #2565
    Planning team/atlas topic/monitoring +3
11. Mimir cost dashboard #2566
    Planning team/atlas topic/monitoring +3
12. Setup Mimir on CAPA installation #3039
    39 of 41
    team/atlas topic/monitoring +2

• https://github.com/giantswarm/giantswarm/issues/26707

Later

• Define cost dashboard
  • current prometheus cost
  • mimir cost
  • prometheus-agent transfer cost
• How to migrate
• Try-it mixins dashboards
• Define/migrate alerting rules
• What happens to SLOs

Related issues

• https://github.com/giantswarm/giantswarm/issues/25272
• Start with Mimir #2027
• https://github.com/giantswarm/giantswarm/issues/27140

[mcbenjemaa]

helm configuration:

https://grafana.com/docs/mimir/v2.5.x/operators-guide/run-production-environment-with-helm/configuration-with-helm/#manage-the-configuration-with-helm

https://grafana.com/docs/mimir/v2.5.x/operators-guide/run-production-environment-with-helm/configuration-with-helm/#example-of-configuration-managed-with-helm

[mcbenjemaa]

Configure Prometheus to write to Mimir: https://grafana.com/docs/mimir/v2.5.x/operators-guide/deploy-grafana-mimir/getting-started-helm-charts/#configure-prometheus-to-write-to-grafana-mimir

[mcbenjemaa]

Monitor Mimir: https://grafana.com/docs/mimir/v2.5.x/operators-guide/run-production-environment-with-helm/#monitor-the-health-of-your-grafana-mimir-cluster

[TheoBrigitte]

No description provided.

[hervenicol]

The "Kubernetes / Compute Resources / Namespace (Pods)" dashboard shows nice info about CPU/RAM usage for the whole mimir namespace. 👍

[TheoBrigitte]

We want to compare Mimir vs Prometheus at installation level

• Build dashboard for Prometheus cost #2430
• Build dashboard for Mimir cost
  • CPU/Memory usage
  • Storage cost
  • Network transfer
  • Number of timeseries
  • Try-it mixins dashboards

Let's aim for something similar as Loki cost dashboard from #2182

[carillan81]

From the hackaton:

• In grafana running in gauss (using prometheus as a data source) I created a dashboard called Mimir cost Estimate similar to Prometheus Cost Estimate . Currently there is no way to get S3 metrics for storage. A posibble follow up would be install YACE from here: https://promcat.io/apps/aws-s3
  https://grafana.g8s.gauss.eu-west-1.aws.gigantic.io/d/d37ffc71-9200-4f0b-8f9a-9e2af9f6b3232/mimir-cost-estimate?orgId=1

• There is another dashboard with less info that puts together Prom and mimir one in front of the other. Prometheus - Mimir comparative https://grafana.g8s.gauss.eu-west-1.aws.gigantic.io/d/dc39e0c1-d4f1-4c8e-b0ab-e1455621f64c/prometheus-mimir-comparative?orgId=1&var-cluster=gauss

• I have changed several panels in the Prometheus Cost Estimate to be more useful:

  • different aggregations
  • using max_over_time so we get the worst number inside the current range
  • fix some typos in descriptions

• Tested some of the mixin dashboards but they will require some tweaking.

[carillan81]

giantswarm/dashboards#305

[TheoBrigitte]

Status update

Mimir currently deployed on gauss via GitOps giantswarm/management-cluster
No multi-tenancy: using anonymous

Metrics

• Data sent via remote write from current Prometheus servers
• Rules are imported within Mimir using Grafana Agent

Alerting

• Mimir Alertmanager deployed
• Alerts available in Grafana via Mimir Alertmanager datasource
• Mimir Alertmanager config deployed manually via mimir tool using current Alertmanager config
• Mimir ruler evaluate rules for alerting

LTS / Grafana Cloud

• Dedicated Prometheus used for rule evaluation
  • remote read from Mimir
  • evaluate aggregation:* (grafana cloud) rules
  • rules split for better performances
  • remote write to Grafana Cloud (currently duplicated in Mimir ruler)

TODOs

• How to keep external labels
• Fix source link in alerts
• Meta monitoring
  • Mimir
  • Grafana Agent
  • Dedicated Prometheus
• Use different S3 bucket for Mimir ruler & Mimir alertmanager config
• Silence-operator: add support for multi-tenancy, to configure silence per tenant
• Define migration path

[TheoBrigitte]

• How to debug Prometheus targets ?

TODOs

• Monitor Mimir, Grafana Agent
• Fix alert source link
• Add new alert templating for Mimir
• Scrape old targets via scrape config
  • Keep old Prometheus to scrape targets via scrape config

Deploy steps

• Switch prometheus-agent remoteWrite from old Prometheus to Mimir
• Stop rule evaluation on old Prometheus

[QuentinBisson]

@TheoBrigitte how about we start by writting down a RFC?

[QuentinBisson]

The discovery phase is over, the epic can be found here #3039. Closing this one