There is a conflict between control planes which are reading from the same topic called "control_plane"

[mikhailsmd]

There is a significant issue involving a conflict between control planes that are reading from the same Kafka topic named "control_plane." This problem occurs irrespective of the deployment type, whether "API-Server" or "Manual."

Currently, there is no option available to change the name of the topic in the message queue used by the Control Plane server. This limitation results in an odd and undesirable behavior wherein one control plane may wait for a message that has already been consumed by another control plane. Consequently, this leads to timeout errors as the control plane is unable to provide a timely response.

Steps to Reproduce:

Deploy multiple control planes using either "API-Server" or "Manual" deployment methods.
Ensure all control planes are configured to read from the Kafka topic "control_plane."
Observe the behavior of the control planes and note any timeout errors or unexpected message consumption patterns.
Expected Result:

Each control plane should independently consume messages from its own specifically designated topic to prevent conflicts and eliminate timeout errors.

Actual Result:

All control planes read from the same topic, resulting in message consumption conflicts and timeout errors.

Proposed Solution:

Introduce a configuration option that allows users to customize the topic name used by each control plane server to prevent conflicts and ensure smooth communication.

[masci]

Thanks for the thorough report! This is definitely something we have to fix, putting in my backlog.