Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use new "created timestamp" for counters and histograms #538

Open
pedro-stanaka opened this issue Feb 19, 2024 · 1 comment
Open

Use new "created timestamp" for counters and histograms #538

pedro-stanaka opened this issue Feb 19, 2024 · 1 comment
Assignees

Comments

@pedro-stanaka
Copy link
Contributor

Summary

One of the issues of using this exporter outside of the sidecar model is that it might generate false values when using rate(), specially for counters and (conventional) histograms (which are also counters at the end). Due to how the function itself works and how data is incremented inside the exporter.

Lenghty explanation - click to expand For example, an application could send 5 increments to the metric `myapp.http.requests.total` before the exporter is scraped for the first time, this will result in a scrape like:
myapp_http_requests_total{} 5

If the metric stays with that same value for a while, when using rate() function over it will result in 0 because the value never changed, when in reality it was not existent before so it went from 0 to 5 because the pod just gone up.

More on this problem, in the excellent talk from [Bartek and Arthur](https://www.youtube.com/watch?v=nWf0BfQ5EEA&t=9s).

(Very) Recently, Prometheus started to accept zero-samples to tell it when a counter started to exist and therefore be able to properly calculate rates on series that are created like non-zero values.

Would be really nice to start preparing the exporter for this to work and we have more consistent metrics between Statsd and Prometheus. I would be willing to provide a PR in the near future if the maintainers think there is value in it.

@matthiasr
Copy link
Contributor

Absolutely, please do!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants