Goal

Try out mimir guide, not for production use.

Changes from Guide

1. Deployed Prometheus in the same network and used Remote write endpoint for Prometheus - http://mimir:9009/api/v1/push. Kept seeing Failed to send batch, retrying + lame referral error when endpoint is not reachable.
2. Overridden Grafana password to be kidding.
3. Grafana datasource in yaml.
4. Sample Application is in the folder app and it keeps generating one metrics and you can generate a metric with 10 labels
5. No LB

Learnings

1. Grafana Mimir is an open source, horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
2. components: distributor, compactor, ingester, querier, query-frontend and store-gateway
3. Component split helps to horizontally scale and allows maintenance with zero downtime
4. Support multiple object store implementations. You may want to keep data in pvc
5. check possibilities of using external Memcached
6. Mimir doesn’t do deduplication by default, like when Prometheus with 2 replicas.
7. Cortex seems to miscalculate quorum when one ingester is unhealthy.