Guidance for Storage options for running Observatorium with operator.

[bwplotka]

cc David (:

Can you share a little about problems you are having and existing configuration?

[david-martin]

Sure.

We're running observatorium via the operator.
There's a number of clients sending metrics & logs into observatorium.
We're also using minio for storage.

Currently, we hit problems with minio storage filling up and resort to using the minio client mc to clean up old data and free up space.

e.g.

mc  rm  --older-than 2d -r --force  minio-obs/thanos

So, maybe the ask here is already possible and it's a knowledge gap on my part (and maybe it's something that could be made simpler through abstraction?).

The ask is a way to easily configure a combination of the max retention by data size and by date.
For example, never go above 50G storage.
Also, never keep data more than 7 days old.

So ideally you will always have 7 days data, but sometimes you won't because your storage is maxing out and needs to be pruned (hopefully automatically)

Also, it would be great to configure this via observatorium regardless of the underlying storage implemention (minio)