Question About query performance of Thanos store gateway

[vansun2022]

Hi,

I am new to Thanos and am eager to learn more about its implementation. I have come across some resources, but I still have a few questions regarding how to improve query performance of Thanos store gateway.
I am using Thanos helm chart to deploy Thanos , and sidecar mode with Promethues. At first, My store gateway have memory pressure when query some metrics. So I increased the memory limitation of store gateway from 12G to 16G. But sometimes it would also reach to 16G when I tried to query metrics in 180d. So I decided to add replica due to memory limitation on one node. I scaled storegateway replica from 1 to 2, and they were separately assigned to 2 node . But when I load test Thanos by sending multiple query. Only replica 0 have resource pressure and replica 1 is calm even resource of replica 0 have reached to limitation.
That's weird for me. In my imagination, if the resources of replica 0 have reached their limit, the query should be automatically routed to replica 1. I don't know how this loadbalance worked. Then I found store gateway is Statefulset kind.
I want to know why store gateway use Statefulset kind (make sure pod always connected to same persistent volume?). If the Statefulset kind related to the this loadbalance situation. And anyway I can take to improve my query performance beside expand node resource or store gateway resource.

If anyone could provide some guidance or point me towards helpful resources, I would greatly appreciate it!

Thank you for your time and assistance.

Best regards.

[dosubot[bot]]

Hi @vansun2022,

I'm here to help you with your Thanos store gateway query performance issue. Let's work together to get this sorted out!

Looking forward to assisting you further.

To improve query performance in Thanos Store Gateway and address your concerns:

1. Load Balancing: Thanos uses a static entry with a load balancer, like a Kubernetes Service, to distribute queries across Store Gateway replicas. Each replica is stateless, so requests can be handled independently. If only one replica is under pressure, ensure your load balancer is correctly configured to distribute traffic evenly ^[1].

2. StatefulSet Usage: StatefulSet is used to provide stable network identities and persistent storage for caching, which helps improve restart times and performance. It doesn't directly affect load balancing but ensures each pod has consistent storage ^[2].

3. Performance Optimization:

   • Caching: Optimize caching settings to reduce memory usage.
   • Autoscaling: Ensure autoscaling is configured to handle load spikes.
   • Tracing and Debugging: Set up tracing and enable debug logging to identify bottlenecks ^[3][4].

4. Scaling: Consider increasing the number of replicas and ensure they are evenly distributed across nodes to handle more queries simultaneously.

These steps should help improve your query performance and better distribute the load across your Store Gateway replicas.

To continue talking to Dosu, mention @dosu.