mimir-querier needs a lot of resources

[QuantumEnigmaa]

Lately, we've been continuously paged by the MimirHPAReachedMaxReplicas alert, mostly for the querier component. Currently this component has a maxReplicas number of 10 with its hpa and a default resources requests of 100m for CPU and 500Mi for the memory.

According to the following per-pod resources usage, CPU is fine but memory could use a little increase :

From our current understanding, the querier needs to scale whenever someone uses its installation's grafana, which would mean that it should scale down during the night. Because of this, I would advocate to increase the maxReplicas to 15 and slightly increase its default memory requests in the shared-configs repo (i;e something like 650~700Mi) to avoid having to increase the former forever.

[hervenicol]

Looks like the RAM usage is quite stable over time, as you can see over 10 days here:

I've increased its RAM reservations to 1GB on a few installations:

• alba
• alligator
• armadillo
• avocet
• wallaby

Let's see in a while if it stabilises at 512MB or if it grows up to 1GB.

We could also do a bit of profiling (with pprof) to try to understand what uses this RAM.