how to achieve multi-tenancy that scales?

[oronsh]

hello,
first of all, thank you for this amazing project!
i'm trying to understand how to scale thanos for say thousands of teams?
from what i read it could be achieved by "soft tenancy".
so my question is, is it required to have each team has its own prometheus TSDB (which would probably be expensive)?
I would really like each team to write and query data from S3, is it achievable with thanos?
more precisely what i would really want to do is using the Receive component for remote write and then allowing teams to query their data, it could be thousands of teams, can thanos support that kind of scale without each team has its own TSDB?

Thanks a lot!

[bwplotka]

Hi, Thanks for raising this question!

I assume you mean if each team should have its own scraping Prometheus, right? There couple of options, but hard tenancy for the scraping part is not a must. You can totally configure Prometheus to send multiple remote-writes per tenant, each with a special tenant header that will tell receiver what tenant it is. See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write - note that you can add multiple of those sections per tenant on single Prometheus.

You will be able to do the same on Prometheus agent, plus similar exists on Grafana Cloud Agent too.

Hope that helps!