Gaps in metrics
#6322
Replies: 2 comments
-
Hey, thanks for reaching out. This can be a problem with both clustering and hashmod sharding; although it may be a little easier to see in clustering due to the fact it can scale dynamically. I've opened https://github.com/grafana/agent/issues/6333 to track a solution to this. |
Beta Was this translation helpful? Give feedback.
0 replies
-
thanks @tpaschalis |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello,
I have a cluster of agents that scrapes a bunch of metrics via Prometheus ServiceMonitor CRD.
![image](https://private-user-images.githubusercontent.com/34036079/302804465-f99a3dad-0878-45b7-8d91-a68a33ef77ce.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxNTM2NTEsIm5iZiI6MTcyNjE1MzM1MSwicGF0aCI6Ii8zNDAzNjA3OS8zMDI4MDQ0NjUtZjk5YTNkYWQtMDg3OC00NWI3LThkOTEtYTY4YTMzZWY3N2NlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDE1MDIzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTA3ZjQwMGQzOTFiNmExYmFkODM0N2Y2NjVjMjRiNTliYTE2NjFjZWFlMjc3MzdjOWJmMmVmMzFjZjEwNTUzMTMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.ai6aKa-sFQgLJuGWFFvhXpIndkqY1G87N1r8hazZ5bs)
![image](https://private-user-images.githubusercontent.com/34036079/302804302-6badd58f-54a9-4056-9dcf-2f019c9967dd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjYxNTM2NTEsIm5iZiI6MTcyNjE1MzM1MSwicGF0aCI6Ii8zNDAzNjA3OS8zMDI4MDQzMDItNmJhZGQ1OGYtNTRhOS00MDU2LTlkY2YtMmYwMTljOTk2N2RkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA5MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwOTEyVDE1MDIzMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTE3NzljNDUyZGM3NDY0YmI5N2NmYmUwM2IzOGZiNjM1MTFhYzg4MmE1NzBjZjExZGNiYzhjMDQ5YWVlMDZhNzkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.yK-mjBPtAdiBtCW-6CpCYSiCXvgu5AHlaSvQyQVlw48)
And I'm investigating the cause of the gaps in some of my metrics. It looks like they are happening when a new agent is starting and picking up the task to scrape these metrics or the agent who scraped these metrics died.
here is the metric that I investigated:
the gap is 5 minutes
and here is a graph where we can see that a new agent was started at that time:
There are no errors in the logs.
So, the question is, are there any configuration options or arguments that declare how the agent's cluster handles new joins or leaves? Or probably any other suggestions on how to ensure high availability for metrics?
Beta Was this translation helpful? Give feedback.
All reactions