Prometheus running out of disk space

[vanzod]

Prometheus fails when ingesting metrics from large number of targets due to running out of disk space on the scraper VM.
Error from Prometheus container log:

ts=2024-04-02T21:10:41.628Z caller=db.go:982 level=error component=tsdb msg="compaction failed" err="compact head: persist head block: 2 errors: populate block: write chunks: preallocate: no space left on device; sync data/01HTGACEHA5PXDFSSPH2SBEB1R.tmp-for-creation/chunks/000001: file already closed"