fix: Too many requests to GET /v1/servers/{id}
#661
Comments
|
This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs. |
lukasmetzner
added a commit
that referenced
this issue
Dec 16, 2024
This includes metrics about internal operations from `k8s.io/cloud-provider` like the workqueue depth and requests to the Kubernetes API. This metrics were already exposed on `:8233/metrics` but this was not documented or scraped. This commit now uses the same registry for our metrics and the Kubernetes libraries, and also exposes them on both ports for backwards compatibility. Besides having all data available, this will also help us with debugging #661. Co-authored-by: Lukas Metzner <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
TL;DR
In some situations hcloud-cloud-controller-manager starts to spam
GET /v1/servers/{id}for a subset of the nodes in the cluster every few seconds.Expected behavior
I would expect hccm to only send a single request per
--node-status-update-frequency(defaults to 5 minutes).Observed behavior
We get reports from customers that exhaust the rate limit and we can see that hcloud-cloud-controller-manager sends > 1 req/s for
GET /v1/servers/{id}. The default rate limit is 1 request per second.A restart of the pod fixes the behaviour.
Minimal working example
We are not sure how to reproduce this yet.
Log output
No response
Additional information
Looking at some request logs, this seems to affect even very old versions of HCCM (>2 years).
The text was updated successfully, but these errors were encountered: