Large MTUs may cause issues joining a cluster

[askeydeteque]

I was having issues with having an 8th node joining a cluster. I would have the nodes join one at a time, and once the 8th node tried to join I would get error messages that is could not, with not much description of why.
Jul 20 16:28:54 REDPANDA-01 rpk[5036]: INFO 2021-07-20 16:28:54,827 [shard 0] cluster - members_manager.cc:432 - Processing node '8' join request
Jul 20 16:28:54 REDPANDA-01 rpk[5036]: WARN 2021-07-20 16:28:54,827 [shard 0] cluster - members_manager.cc:448 - Error adding node {id: 8, kafka_advertised_listeners: {{:{host: <REDACTED>, port: 9092}}}, rpc_address: {host: 10.130.0.18, port: 33145}, rack: {nullopt}, properties: {cores 8, mem_available 0, disk_available 0}, membership_state: active} to cluster - raft::errc::configuration_change_in_progress

I tried joining them in different orders and the problem did not seem to follow a specific node, but the 8th node to join never could. After some troubleshooting we moved the MTU from 9000 to 1500. Immediately after this the nodes did not have any issues joining the cluster.

[emaxerrno]

thanks for the Flight Rule @askeydeteque !!