Peer Discovery issues using 3.13.x in MS Azure Service Fabric

[abeckham7305]

I'm running an application in Microsoft Azure Service Fabric with three nodes, using RabbitMQ as my message broker. I have the nodes in rabbitmq.conf configured using the simple classic_config option. I specify the three nodes there as follows:

cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
cluster_formation.classic_config.nodes.1 = rabbit@node1
cluster_formation.classic_config.nodes.2 = rabbit@node2
cluster_formation.classic_config.nodes.3 = rabbit@node3
cluster_formation.node_type = disc

This worked fine for me in 3.12.x versions of RabbitMQ. The nodes would join the same cluster whether I was upgrading the RabbitMQ version or installing it new.

In version 3.13.x this doesn't work for me anymore when installing new. If upgrading it works fine since the nodes have already been discovered and stored in the data files. I believe the issue is with recent updates to how peer discovery works. I don't understand the code enough to figure this out myself. It seems as if the new peer discovery method is starting a hidden node to perform the clustering and this hidden node is using the default erlang cookie for the user instead of the one I am specifying as a parameter in Service Fabric to pass to RabbitMQ. I'm passing this cookie value using this method for my parameter:

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-setcookie abcd-efgh-1234-5678-etc"

The errors I keep getting are like this, which I read are typically related to mismatched erlang cookies:
** Connection attempt from node 'rabbit-930-4988@node2' rejected. Invalid challenge reply. **
** Connection attempt from node 'rabbit-706-6372@node3' rejected. Invalid challenge reply. **

Any thoughts on whether this is a bug or if I should be doing something differently?

[lukebakken]

Thank you for your comprehensive report. Indeed, the new peer discovery code does not check RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS for -setcookie, but it DOES use RABBITMQ_ERLANG_COOKIE in the peer discovery code.

So, I'm pretty sure your issue will be resolved if you remove -setcookie from RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS (or delete that env var if it's only used for the cookie), and, instead, use RABBITMQ_ERLANG_COOKIE.

Please let me know how this works.

[lukebakken]

PR to allow using RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS: #11654

[abeckham7305]

Thank you for the quick reply! I'll test your recommendation out today and let you know the results.

[abeckham7305]

Using RABBITMQ_ERLANG_COOKIE instead totally worked! Thanks again!!

[lukebakken]

Thank you for the follow-up.