fix(kds): detect properly hanging stream for the same zone

[lukidzi]

Motivation

In a multi-zone architecture, we run a global control plane and multiple zone control planes. Each zone can have multiple instances running, but only the leader is responsible for communicating with the global control plane.

Currently, if there is an issue with a zone or a proxied stream that does not properly detect the connection closure, we might end up in a situation where a zone control plane restart goes unnoticed. This could result in multiple open streams from a single zone to the global control plane. Since the health check only verifies the zone name and tenant ID, a newly connected instance of the control plane makes all previous health check detectors pass.

This issue occurs when a gRPC client and server, both configured with Keepalive, communicate through a Gateway that is also configured with the gRPC protocol. We can reproduce it with the following setup:

Zone → Gateway (gRPC) → Global

By introducing packet loss on the Zone, we can observe the following log messages in the control plane:

keepalive ping failed to receive ACK within timeout
However, despite this failure, the connection remains open in the control plane, leading to unnecessary resource allocation. This suggests that the Gateway might be misconfigured. In such cases, we should implement safeguards to prevent hanging connections.

Implementation information

We could track the last connection time by recording it in an event and later checking if the same client attempts to reconnect. If a reconnection occurs, we cancel the old stream.

To handle scenarios where multiple instances of the global control plane are running, we will store this information in zoneInsight. Additionally, we will use a periodic verification mechanism (zone watcher) to ensure the recorded time remains accurate.

[github-actions]

Reviewer Checklist

🔍 Each of these sections need to be checked by the reviewer of the PR 🔍:
If something doesn't apply please check the box and add a justification if the reason is non obvious.

• Is the PR title satisfactory? Is this part of a larger feature and should be grouped using > Changelog?
• PR description is clear and complete. It Links to relevant issue as well as docs and UI issues
• This will not break child repos: it doesn't hardcode values (.e.g "kumahq" as an image registry)
• IPv6 is taken into account (.e.g: no string concatenation of host port)
• Tests (Unit test, E2E tests, manual test on universal and k8s)
  • Don't forget ci/ labels to run additional/fewer tests
• Does this contain a change that needs to be notified to users? In this case, UPGRADE.md should be updated.
• Does it need to be backported according to the backporting policy? (this GH action will add "backport" label based on these file globs, if you want to prevent it from adding the "backport" label use no-backport-autolabel label)

[lukidzi]

Should we implement a feature flag to disable it? Please vote

[lobkovilya]

The PR makes sense. I'm a bit worried if it's going to be a mess when on each connection at least 6 concurrent processes are going to try to update the same ZoneInsight resource. If at least one of them fails others are going to be canceled (and retried on the client).

As it's potentially can be a regression I'd hide this behind the feature flag and disable by default.

[lahabana]

Isn't the retry below just fine?

[lukidzi]

It's fine, but it doesn't solve the issue with the first request. It only improves the probability that the first request won't hit a conflict.