Recommended method of cluster management

[stratacast]

I've been running ra for a while and ran into an issue where I couldn't get a node to reconnect until I cleared out the .segment files in my radata folder. The partitioned node seemed to run happily all alone while the rest of the cluster did its thing. This incident lead me to see that I don't think I am doing this clustering thing quite right. I ra:start_cluster/4 a 3 member cluster. I've had it once where I had to shut down the cluster entirely to get everything reconnected, along with this segment file clearing (which may have been the solution previously).

With some of this jank in my code, I think I am starting my cluster in an unreliable way. So, is the right way to start as a single node then add members? When a node disconnects to I remove it, then wait for an event like when the server shows up in the nodes() list and add the member back? There's a lot of ways to skin the cat with ra. What do you all suggest?

[michaelklishin]

@stratacast I don't see a connection between the two. Chances are your node was out of disk space because the log wasn't truncated due to how you use Ra, we do not know what specifically.

Or maybe your cluster did not have a majority of replicas online back in that moment, and by the time you have decided to "reset" a node, it was.

ra:start_cluster/N is what RabbitMQ uses, although not across the board. Stream coordinator uses this function, for example.

It's the monitoring of Raft-related metrics and the free disk space that you likely need, not another function to start a cluster.

[stratacast]

Understood. In that case, this is still a good rough template of what is a good way to get a cluster going and keep it running?

https://github.com/rabbitmq/ra-kv-store/blob/main/src/ra_kv_store_app.erl

I have something similar to the above, but looking at my code now vs this I see an improvement I can make.

I have plenty of disk space, so perhaps there was an issue with majority replicas and some strange partitioning in general or maybe some corrupted data. Not being able to start the node to rejoin, was removing the .segment files an appropriate action?

[michaelklishin]

The K/V store is just an example (and a way to run Jepsen tests against Ra, since Ra is a library).

Khepri, stream coordinator and quorum queues in RabbitMQ are three open source examples of how Ra clusters are formed. There isn't just one true way of doing it, e.g. Ra does not dictate how/if you perform peer discovery.