Grafana Agent with remote write receiver

[naveen-adisesha]

Hello,
If the remote write destination is down for several hours, with the default configuration of grafana agent, how long can I expect the downtime to be, if I have provided sufficient disk space for the agent.

Will I be able to manage(all data available at receiver) 1 day network down by providing sufficient disk space ? What flag should I use to achieve this ?

Thank you for any information in this regard.

Regards
Naveen A

[rfratto]

Hi @naveen-adisesha, sorry for the long delay on getting back to you. The downtime tolerance depends on how frequently the WAL is truncated. As of 0.13.0, our default is 60 minutes, which gives you up to 2/3rds of that in downtime tolerance. I'm working with the Prometheus team on changing this, because it's confusing. The doc of my proposal (which explains more about downtime tolerances) is here.

If you want to tolerate 24 hours of downtime, you'll probably want to set your wal_truncate_frequency in the Agent to 48h. How much storage that'll use depends on your metrics throughput, but it would likely take up more storage than Prometheus TSDB over an equivalent time range since the TSDB stores data compressed.

[naveen-adisesha]

Thanks a lot for the reply. Will check the document.