Add ability to optionally expose request timestamp to metrics?

[ahurtaud]

Hello,
I would like to discuss if it would be accepted to add a feature flag to add timestamp (request time) to the exposed metrics. I dont think this is possible right now.

Basically, we have some network devices (Juniper) which can get ~4minutes to reply to some SNMP data and is quite heavy computation to the juniper devices.
To avoid harming too much the device cpu (it is the management cpu but still, we dont like them hitting 100% usage for 4 minutes), we have put an nginx cache so that our second prometheus HA get the same data.

Because the nginx cache TTL validation is based on the reply timestamp, it may happen that we get some counter "decrease" because our HA pair get updated values with timestamps set before its pair.

Optionally exposing the "request timestamp" and using honor_timestamp: true seems to fix our usecase. As both HA pair will ingest data on aligned timestamps whichever it hits the cache or not.

WDYT?

[SuperQ]

Sorry, this is not something we plan to implement. Timestamps in Prometheus are meant for importing data from Federation. There are a number of pitfalls that you will run into that cause problems if you try and use timestamps. The timestamp Prometheus attaches to the scrape is the start of the scrape, which represents when Prometheus sees the data.

For Juniper devices, there are a number of things you can tune to improve scraping that should let you poll at a more reasonable rate without causing issues. See the Juniper best practices documentation.

For example, you should enable device-side caching in JunOS with set snmp stats-cache-lifetime X. Where X is your desired scrape interval (I sometimes do scrape interval - 1 second).

Second, reduce your scrape scope and shard the scrape so that it doesn't try and do the whole thing at once. For example, this is my JunOS generator config.