Try out mimir guide, not for production use.
- Deployed Prometheus in the same network and used Remote write endpoint for Prometheus -
http://mimir:9009/api/v1/push. Kept seeingFailed to send batch, retrying+lame referralerror when endpoint is not reachable. - Overridden Grafana password to be
kidding. - Grafana datasource in yaml.
- Sample Application is in the folder app and it keeps generating one metrics and you can generate a metric with 10 labels
- No LB
- Grafana Mimir is an open source, horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
- components: distributor, compactor, ingester, querier, query-frontend and store-gateway
- Component split helps to horizontally scale and allows maintenance with zero downtime
- Support multiple object store implementations. You may want to keep data in pvc
- check possibilities of using external Memcached
- Mimir doesn’t do deduplication by default, like when Prometheus with 2 replicas.
- Cortex seems to miscalculate quorum when one ingester is unhealthy.