Cortex Ingester Memory Usage Ever Increasing

[akshat-zopsmart]

Using cortex v1.13
We're seeing cortex ingester memory usage continuously increasing. While it decreases slightly in between, overall it's increasing gradually.
Right now we have ~346GB memory utilisation (100 pods).
Memory usage for the last 6 months can be seen below

Not able to figure out how to reduce the memory utilisation

[friedrichg]

can you include the same dashboard with sum(cortex_ingester_active_series) ?. That should give us information about the problem in relation to active series.

If you see memory increasing when active series increase, then what you are seeing is expected. More series require always more memory.

[akshat-zopsmart]

Cortex metrics are not being scraped by prometheus setup on the cluster, so we don't have the cortex_ingester_active_series metric graph available.
As per the /distributor/all_user_stats end point, current active series is 20M for all tenants included on this (non-prod) environment. On the prod env, it's 4.5M series.
Memory utilisation for ingesters on prod env is ~45GB, which is proportionately much lower than the non-prod env.

Also, is there some "rule-of-thumb" ratio of metrics count vs memory usage which can be used to plan the infra sizing?

[friedrich-at-adobe]

the rule of thumb for ingester memory usage is mentioned on https://cortexmetrics.io/docs/guides/capacity-planning/
the all_user_stat endpoint gives visibility for current load, but that can change in a long period of time.

There are many other internal metrics cortex exposes, if you don't scrape them you won't be able to have dashboards to monitor cortex usage or basic alerts. See https://github.com/cortexproject/cortex-jsonnet#monitoring-for-cortex

[akshat-zopsmart]

Thanks @friedrich-at-adobe
We're using cortex helm chart to setup cortex on the kubernetes cluster. Monitoring has been enabled for each of the component as well as top level (config.serviceMonitor.enabled) but it's not working for some reason. Any pointers on how to fix that?