Master nodes with unpredictable names

[rocketraman]

Another problem I've found running clusters with this setup: if all the master nodes go down for some reason (I had this issue with my masters after applying an updated Deployment), the client/data nodes do not automatically recover.

This is because the client/data nodes are still looking to connect to the old master node names, none of which exist any longer. I don't know if this is a bug in the zen discovery protocol, or expected behavior, but the client/data nodes simply do not recover without a restart.

Its possible that this could be solved by giving the master nodes a stable identity, which in turn could be achieved by making the master nodes a StatefulSet. Even though they actually require no state in terms of persistent disk, their name is the "state" being persisted.