[Thanos Store] Open file limit reached. What are our options

[erikeverts]

Hi,

we are running HA thanos in a cloud foundry environment with each thanos component as a seperate app. On thanos store we had reached the open file limit earlier, and then deployed thanos compactor to reduce the number of files in object storage. This worked for a short while, and now we are hitting the open file limit again. We don't have the option of increasing the maximim open file descriptors since this limit is controlled by cloud foundry settings. The current setting is just over 16k. Below are out configs for thanos store and thanos compactor. Is there anything we misconfigured to hit this limit so fast. Currently there are about 15-20 prometheus instances sending data to thanos receiver and that data is stored in object storage. We have hit this limit within 3 months. Plans are for hundreds of prometheus instances to send their metrics to thanos so hitting the limit so fast with such small number of prometheus sender is problematic. Thanos should be able to handle very large amounts, so possibly there are some things configred wrong on our end. Below are the configs for both thanos store and thanos compactor.

The s3 bucket currently contains about 230GB of data and has around 110000 objects

[EDIT] Adding version info
Thanos store 0.21.1
Thanos compactor 0.22

Thanos store config

    --http-address=0.0.0.0:9090
    --data-dir=/prometheus
    --index-cache-size=500MB
    --chunk-pool-size=500MB
    --objstore.config-file=/sidecars/etc/bucket_config.yaml 

bucket_config.yml

type: S3
config:
  bucket: "redacted"
  endpoint: "s3-external-1.amazonaws.com"
  access_key: "redacted"
  insecure: false
  signature_version2: false
  secret_key: "redacted"

Open file limit

~ # ulimit -n
16384

Thanos compactor config

    --http-address=0.0.0.0:9090
    --data-dir=/compactor
    --objstore.config-file=/sidecars/etc/bucket_config.yaml
    --consistency-delay=60m
    --delete-delay=1h
    --log.level=info
    --log.format=logfmt
    --retention.resolution-raw=0d
    --retention.resolution-5m=0d
    --retention.resolution-1h=0d
    --compact.enable-vertical-compaction
    --wait
    --wait-interval=5m

Error messages from thanos store

2021-09-09T11:17:14.16+0200 [APP/PROC/WEB/1] OUT 2021/09/09 09:17:14 http: Accept error: accept tcp 0.0.0.0:9090: accept4: too many open files; retrying in 5ms

2021-09-09T11:17:12.15+0200 [APP/PROC/WEB/0] OUT level=warn ts=2021-09-09T09:17:12.152488457Z caller=bucket.go:486 msg="loading block failed" elapsed=113.243008ms id=01FEQ548SVC1HMX9WM233WDERR err="create index header reader: write index header: new binary index header writer: open /prometheus/01FEQ548SVC1HMX9WM233WDERR/index-header.tmp: too many open files"

[yeya24]

Is there a way to increase the open files limit in CF?
If that's not possible then you can try to shard Thanos stores by hash or by time.

[erikeverts]

Thanks for the reply. It is indeeed not possible to increase the limit. We have done sharding now, but this makes only two weeks of data available. So for older time periods I would have to create a new store with another time period sharded right? This doesn't make it very scalable if you have to create new stores for every two weeks or so of data. Are there any other option that we could consider. Also, what is usualy a time frame for data to be available to query. I'm guessing with this setup it's not usual to have data be available forever?

[yeya24]

Sharding by time is one way. Another way is to shard data by block IDs.
And why time sharding makes only 2 weeks of data available?