-
Notifications
You must be signed in to change notification settings - Fork 171
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
alloy metrics pod crashes with alloy v1.2.1 when using prometheus.operator.servicemonitors config #1349
Comments
Running into to the same, downgrading to |
We encountered the same issue and can confirm that it runs fine if updating from In addition, with |
Hey, thanks for reporting this bug :)
This is actually the fix for this bug, it's not yet in v1.2.1. (the panic above indicates that it's trying to access the first value in the slice but the slice is empty. The fix is checking for the slice length to avoid this). |
Oh I somehow missed that it is not yet in 1.2.1 |
Hi @wildum , just want to follow up on the bug, do you think the fix is on track to release this week? Thanks. |
Hey, the release is slightly delayed, we are aiming for next Monday (5th of August) |
Thanks @wildum , does v1.3.0-rc.0 have it though? I was thinking about give it a try. |
it does yes :) Feel free to try it, keep in mind that this is a release candidate that may contain bugs. We test it internally before we release a stable version |
I tested v1.3.0-rc.0, just to provide some data point, it did not give index out of range error anymore. However, it still fails to form the cluster. It is still showing the same behavior in the dev image.
|
@thampiotr do you know why this happens? |
It happens because Alloy fails to resolve the provided cluster join peer addresses at startup:
In previous versions, there was another bug where at startup a failure to resolve the cluster join peers would not result in failure and errors would not be logged. This is undesired behaviour, because such misconfiguration can result in all instances starting their own cluster and trying to take on all the work, resulting in duplication and likely resource exhaustion. We want to fail fast and fail hard in this scenario, hence you get this error. This behaviour was changed here. Try to verify that the name |
Thanks @thampiotr. Looking back, in I found a way to fix the DNS issue by adding this to the service-cluster.yaml: |
This is now working with version |
Version 1.3.1 addresses this: https://github.com/grafana/alloy/releases/tag/v1.3.1 |
What's wrong?
We are using prometheus.operator.servicemonitors in our config for alloy metrics pod in kubernetes and with the update to version 1.2.1 new pods crashing directly. The pods do not crash if an existing pod gets updated from alloy 1.1.1 or 1.2.0 to 1.2.1.
Maybe this has to do with this recent code change
Steps to reproduce
Deploy a new pod don't update an existing pod, because then the issue does not occur.
System information
GKE 1.27.11-gke.1062001
Software version
Grafana alloy 1.2.1
Configuration
Logs
The text was updated successfully, but these errors were encountered: