How to preserve undelivered telemetry data during outages or restarts?

[KieranP]

Hello. What is the best way to preserve undelivered telemetry during outages or restarts, such as when the exporter destination is offline or between collector restarts?

Background: We have an EC2 instance running Signoz. And we have 20+ EC2 instances running the collector. The collector is setup to send data to Signoz. This all works fine under normal running conditions.

Issue: We recently updated Signoz and lost about 2 minutes of metrics/traces/logs during the update. It seems that the collectors may have dropped the data because it couldn't connect to the exporter destination.

Questions:

1. When the exporter destination is offline, are the collectors meant to keep undelivered metrics/traces/logs and retry them? And if so, are there settings to tweak how much data to keep, retry rates, etc etc?
2. Occasionally also we will update the collectors. When we do so, what is the best way to ensure that unsent telemetry data isn't lost? (i.e. is there a way to write unsent data to disk that the collector then reads on next startup?).

Thanks for any help given.

[KieranP]

After much searching, finally found this blog post which helped: https://incerto.in/blog/leveraging-persistent-storage-for-opentelemetry-otlp-exporters