Support for a RabbitMQ cluster without a load balancer

[nesb1]

A RabbitMQ cluster can have multiple nodes, but rstream Producer and Consumer classes have host and port arguments in __init__ only for one node. I know that rstream support cluster dicovery and after discover rstream will produce messages to leader and will consume messages from replicas. But when I want to use rstream with cluster without load_balancer_mode I need to choose between nodes to initial client. When the application starts, the selected node may not be available but others are available and rstream will not work. Also when node available, consumer or producer started and leader node brokes after some time and rstream cannot use it default client to discover cluster and resolve a problem.

As user of rstream I want to pass list of nodes (host and port) to Consumer and Producer and rstream will resolve all nodes disconnection problems to stay reliable. Related issue - now consumers cannot reconnect automatically, for this you need to define a special callback on_close_handler. It looks like in order to teach stream to work reliably with a cluster without a balancer, this problem also needs to be solved.

[DanielePalaia]

Hi @nesb1,

Yes I think this is a very useful functionality to have. Thanks to have raised this.

To keep the compatibility with the existing users of the library we can let the "host" parameter of Producer and Consumer to be both a str or a list of str.

During the connection of the default_client we could loop over all the nodes till we get a connection.

I don't think that the fact that Consumer doesn't support automatic reconnection is a big issue.
Usually inside on_close_handler you will reconnect to a stream (using consumer.reconnect_stream) and if the default_client is not connected the client should try to reconnect (reconnect_stream is calling subscribe which is calling create_subscriber which is calling _get_or_create_client).
Also the reconnect_stream always close the default_client:
https://github.com/qweeze/rstream/blob/master/rstream/consumer.py#L503

In case of a reconnection we can reuse the same approach trying to reconnect to the first available server.

I agree with you that it would be nice to have Consumer to reconnect automatically too but maybe is better if possible to keep it in separate issue/PR.

While this functionality is relevant I'm not sure if we will have the bandwidth to implement this shortly so If you like and have time to raise a PR this will be appreciated. Let us know if you can otherwise we will plan to do it when we get time!

[nesb1]

@DanielePalaia The task is quite difficult, unfortunately I don't have free time for such a task right now, but I would like to help, maybe in 1-2 months I will be able to.

[DanielePalaia]

@nesb1 Sure np, in case you are planning to work on it in the future ping us before to avoid working on the same issue in parallel :)