Store Gateway PVCs Fill Up Frequently, Require Recommendations

[hecomp]

Describe the bug
Store gateway PVCs in our Cortex project are filling up frequently, forcing us to repeatedly increase the size of the PVCs. We are seeking recommendations to resolve this issue.

To Reproduce
Steps to reproduce the behavior:

1. Start Cortex (SHA or version)
2. Perform Operations (Read/Write/Others)
3. Monitor store gateway PVCs usage
4. Observe frequent filling up of the PVCs, requiring manual intervention to increase their size

Expected behavior
We expect the store gateway PVCs to have a more stable and manageable growth rate that does not require constant manual intervention.

Environment:
Infrastructure: Kubernetes
Deployment tool: Pulumi

Issue
Store gateway PVCs are filling up frequently, forcing us to increase their size regularly. We are requesting recommendations to manage or mitigate this issue more effectively.

[friedrichg]

store-gateway typically download headers and blocks from s3. If the number of blocks in block storage is increasing, it's expected pvcs for the store gateways need to be increased too

pvcs will increase in size till you have reached the retention configured.

Retention is disabled by default (Meaning infinite retention)

# Delete blocks containing samples older than the specified retention period. 0
# to disable.
# CLI flag: -compactor.blocks-retention-period
[compactor_blocks_retention_period: <duration> | default = 0s]

Let's say you configure one year retention in the compactor:

-compactor.blocks-retention-period=365d

With a healthy compactor, there should be one block added per day per tenant, you would need to wait one year till the system stabilizes in terms of number of blocks .

Till that happens you will need to increase pvc size or what is probably better: increase the number of store-gateway replicas.

Also make sure you have the bucket index enabled for store-gateways as well as the index header lazy enabled

Also, make sure the compactor is correctly working and not delayed, as that can cause the number of 2hour blocks to increase also putting pressure in the store-gateways pvcs