Can mimir be configured to use local SSD and HDD?

[CommanderRedYT]

I have two servers. One is running prometheus and mimir and has a 500GB nvme, the other has plenty of HDDs in a RAID system. The mimir server can reach the data volume via a folder mounted over the network.

I mainly use it to store data of multiple solar inverters, so I need the long-term data (multiple years).

My idea was to configure retention to drop resolution after 1y to ~30 minute values, which is good enough. But also, it would be great if I could configure mimir to store the first year on its host on the fast nvme and push old data to the HDD cluster.

Is this possible?

(My setup is running mimir from the distro packages, meaning no helm / docker involved, so I cannot do fancy cluster stuff)

[deniszh]

That's not how Mimir works, unfortunately. It require cloud object storage. You can use something like Minio for that, but maybe whole architecture will be too complex in your case.

[colega]

multiple solar inverters

How many is multiple? A couple, tens or thousands?

My idea was to configure retention to drop resolution after 1y to ~30 minute values, which is good enough.

If the answer to the response above is tens, I wouldn't care about dropping the resolution at all. Also note that Mimir doesn't the downsampling feature. You should consider having recording rules that perform "downsampling".

But also, it would be great if I could configure mimir to store the first year on its host on the fast nvme and push old data to the HDD cluster.

IMO you're optimizing too early.

Also note that Mimir will cache the query results, you can run a memcached with your nvme as additional storage, IMO that will be a much more efficient solution than using your nvme for raw blocks.

Finally, as @deniszh points out, you should run Minio or something like that. Mimir has to talk to an object-storage-ish API.

[CommanderRedYT]

The number should be scalable. For me nothing would work except something where I can push my data and the last couple weeks are on a SSD and everything older is on HDDs. I only have ~100 GB of SSD (which is also my rootfs, so not gonna let that get filled up completely). This can already be done with software like clickhouse, but that of course cannot really be used like Prometheus can be used.

S3 for just storing this is for me another service that is overhead... But it seems that the only good solution to this problem is not using a SSD and live with slower access time...

[CommanderRedYT]

Sadly another time where it's a software that doesn't work for small individuals without a huge cluster and kubernetes and similar stuff

[colega]

I'll respond in reverse order:

Sadly another time where it's a software that doesn't work for small individuals without a huge cluster and kubernetes and similar stuff

It definitely works, but you need a layer of abstraction on top of the filesystem. Mimir focuses on implementing just one storage API: object storage. If we had to support a distributed filesystem on top of multiple machines with HDDs we'd just be building another Minio, and there's already an awesome project that does it very well: Minio.

S3 for just storing this is for me another service that is overhead.

Is it so big? We have it in our docker-compose for development, it doesn't really ask so much.

But it seems that the only good solution to this problem is not using a SSD and live with slower access time...

Note that Mimir is designed to serve queries from object-storage, your HDD is unlikely to affect your latency in such a drastic way.

What's the pattern of queries you're expecting? What's your latency SLO?

As I also mentioned, running memcached with flashstorage should help your latencies if they're so critical.

For me nothing would work except something where I can push my data and the last couple weeks are on a SSD and everything older is on HDDs.

I think "nothing would work" is a too hard statement. I'm sharing with you the experience of running this software on cloud for thousands of customers with different needs.

The number should be scalable.

Mimir does scale, but every system has to be designed with proper numbers in mind.

The requirements of the deployment can't be the same for a Mimir instance that handles 1M series than for the one that handles 100M series.

---

Now, to summarize: you mentioned you have one server with 500GB of SSD and the other one with plenty of HDD space, then you also mentioned that you have ~100GB of SSD on your rootfs (I assume that's the free portion of the 500GB). You are concerned about your recent queries latency and the SSD usage.

In this case, I would definitely discourage storing 1 week of series blocks on the SSD (but if you really wanted to, you can do that by configuring -query-ingesters-within flag to 1w and -query-store-after to 167h (1w-1h), and the ones related to the block retention in the ingesters). Note that this would also hit your performance as ingester doesn't compact its blocks.

I would suggest you to run a single instance Minio on your machine with HDDs, and also a store-gateway instance there.

On the machine with SSD I'd run the query-frontend, query-scheduler, querier, distributor, ingester and ruler. I'd run them as separate process to make sure that an OOM in a querier caused by a huge query doesn't affect your ingester.

Finally I'd run a memcached instance on the machine with SSD with a couple of GB of flashstorage to cache your query results to fulfill your concerns about the speed of queries of the last week. I'd also increase the cache TTL from the default 7d to something higher.

I hope this helps.