Zookeeper fails after updating to Kafka 2.4

[jgottfriedMSC]

Hello there,
Since the Kafka update to 2.4 I'm trying in vain to deploy the new version on my Kubernetes cluster.
I only changed the image version as it was done here in the last pull request, but I have problems getting Zookeeper to work.

It seems that he is no longer able to resolve the host addresses.

Below you can find the yaml configuration of the statefulset, the configmap and the error messages from the container.

In addition, the replication factor for the pods is 2.

Thanks in advance for the help!

10zookeeper-config.yml

50pzoo.yml

ERROR
[2019-12-30 08:27:11,769] INFO Reading configuration from: /etc/kafka/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2019-12-30 08:27:11,772] INFO clientPortAddress is 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2019-12-30 08:27:11,772] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2019-12-30 08:27:11,780] WARN No server failure will be tolerated. You need at least 3 servers. (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2019-12-30 08:27:11,786] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager) [2019-12-30 08:27:11,786] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager) [2019-12-30 08:27:11,786] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager) [2019-12-30 08:27:11,787] INFO Log4j found with jmx enabled. (org.apache.zookeeper.jmx.ManagedUtil) [2019-12-30 08:27:11,806] INFO Starting quorum peer (org.apache.zookeeper.server.quorum.QuorumPeerMain) [2019-12-30 08:27:11,861] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory) [2019-12-30 08:27:11,910] INFO Logging initialized @904ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log) [2019-12-30 08:27:12,070] WARN o.e.j.s.ServletContextHandler@23e84203{/,null,UNAVAILABLE} contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler) [2019-12-30 08:27:12,070] WARN Empty contextPath (org.eclipse.jetty.server.handler.ContextHandler) [2019-12-30 08:27:12,089] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [2019-12-30 08:27:12,093] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [2019-12-30 08:27:12,099] INFO Local sessions disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [2019-12-30 08:27:12,099] INFO Local session upgrading disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [2019-12-30 08:27:12,099] INFO tickTime set to 2000 (org.apache.zookeeper.server.quorum.QuorumPeer) [2019-12-30 08:27:12,099] INFO minSessionTimeout set to 4000 (org.apache.zookeeper.server.quorum.QuorumPeer) [2019-12-30 08:27:12,099] INFO maxSessionTimeout set to 40000 (org.apache.zookeeper.server.quorum.QuorumPeer) [2019-12-30 08:27:12,099] INFO initLimit set to 5 (org.apache.zookeeper.server.quorum.QuorumPeer) [2019-12-30 08:27:12,152] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase) [2019-12-30 08:27:12,154] INFO Using insecure (non-TLS) quorum communication (org.apache.zookeeper.server.quorum.QuorumPeer) [2019-12-30 08:27:12,154] INFO Port unification disabled (org.apache.zookeeper.server.quorum.QuorumPeer) [2019-12-30 08:27:12,154] INFO QuorumPeer communication is not secured! (SASL auth disabled) (org.apache.zookeeper.server.quorum.QuorumPeer) [2019-12-30 08:27:12,154] INFO quorum.cnxn.threads.size set to 20 (org.apache.zookeeper.server.quorum.QuorumPeer) [2019-12-30 08:27:12,155] INFO Reading snapshot /var/lib/zookeeper/data/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileSnap) [2019-12-30 08:27:12,176] INFO jetty-9.4.20.v20190813; built: 2019-08-13T21:28:18.144Z; git: 84700530e645e812b336747464d6fbbf370c9a20; jvm 11.0.5+10 (org.eclipse.jetty.server.Server) [2019-12-30 08:27:12,209] INFO DefaultSessionIdManager workerName=node0 (org.eclipse.jetty.server.session) [2019-12-30 08:27:12,210] INFO No SessionScavenger set, using defaults (org.eclipse.jetty.server.session) [2019-12-30 08:27:12,211] INFO node0 Scavenging every 660000ms (org.eclipse.jetty.server.session) [2019-12-30 08:27:12,254] INFO Started o.e.j.s.ServletContextHandler@23e84203{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler) [2019-12-30 08:27:12,261] INFO Started ServerConnector@71ba6d4e{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} (org.eclipse.jetty.server.AbstractConnector) [2019-12-30 08:27:12,261] INFO Started @1263ms (org.eclipse.jetty.server.Server) [2019-12-30 08:27:12,262] INFO Started AdminServer on address 0.0.0.0, port 8080 and command URL /commands (org.apache.zookeeper.server.admin.JettyAdminServer) [2019-12-30 08:27:12,267] INFO Election port bind maximum retries is 3 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [2019-12-30 08:27:12,268] INFO My election bind port: /0.0.0.0:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) [2019-12-30 08:27:12,278] INFO LOOKING (org.apache.zookeeper.server.quorum.QuorumPeer) [2019-12-30 08:27:12,279] INFO New election. My id = 1, proposed zxid=0x0 (org.apache.zookeeper.server.quorum.FastLeaderElection) [2019-12-30 08:27:12,282] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:764) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456) at java.base/java.lang.Thread.run(Thread.java:834) [2019-12-30 08:27:12,284] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:774) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456) at java.base/java.lang.Thread.run(Thread.java:834) [2019-12-30 08:27:12,287] WARN Cannot open channel to 2 at election address pzoo-1.pzoo:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) at java.base/java.net.Socket.connect(Socket.java:609) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:620) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:477) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:456) at java.base/java.lang.Thread.run(Thread.java:834) [2019-12-30 08:27:12,288] INFO Notification: 2 (message format version), 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection) [2019-12-30 08:27:12,490] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:764) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:12,490] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:774) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:12,491] WARN Cannot open channel to 2 at election address pzoo-1.pzoo:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) at java.base/java.net.Socket.connect(Socket.java:609) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:12,491] INFO Notification time out: 400 (org.apache.zookeeper.server.quorum.FastLeaderElection) [2019-12-30 08:27:12,892] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:764) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:12,892] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:774) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:12,893] WARN Cannot open channel to 2 at election address pzoo-1.pzoo:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) at java.base/java.net.Socket.connect(Socket.java:609) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:12,893] INFO Notification time out: 800 (org.apache.zookeeper.server.quorum.FastLeaderElection) [2019-12-30 08:27:13,694] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:764) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:13,694] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:774) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:13,695] WARN Cannot open channel to 2 at election address pzoo-1.pzoo:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) at java.base/java.net.Socket.connect(Socket.java:609) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:13,695] INFO Notification time out: 1600 (org.apache.zookeeper.server.quorum.FastLeaderElection) [2019-12-30 08:27:15,295] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:764) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:15,296] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:774) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:15,296] WARN Cannot open channel to 2 at election address pzoo-1.pzoo:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) at java.base/java.net.Socket.connect(Socket.java:609) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:15,296] INFO Notification time out: 3200 (org.apache.zookeeper.server.quorum.FastLeaderElection) [2019-12-30 08:27:18,458] INFO The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, -720899=telnet close, 2003003507=wchs, 2003003504=wchp, 1684632179=dirs, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}] (org.apache.zookeeper.server.command.FourLetterCommands) [2019-12-30 08:27:18,458] INFO The list of enabled four letter word commands is : [[srvr]] (org.apache.zookeeper.server.command.FourLetterCommands) [2019-12-30 08:27:18,497] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:764) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:18,497] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:774) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:18,498] WARN Cannot open channel to 2 at election address pzoo-1.pzoo:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) at java.base/java.net.Socket.connect(Socket.java:609) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:18,498] INFO Notification time out: 6400 (org.apache.zookeeper.server.quorum.FastLeaderElection) [2019-12-30 08:27:24,901] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo: Name or service not known at java.base/java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:929) at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1515) at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:848) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:764) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:24,901] WARN Failed to resolve address: pzoo-1.pzoo (org.apache.zookeeper.server.quorum.QuorumPeer) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:797) at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1505) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1364) at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1298) at java.base/java.net.InetAddress.getByName(InetAddress.java:1248) at org.apache.zookeeper.server.quorum.QuorumPeer$QuorumServer.recreateSocketAddresses(QuorumPeer.java:194) at org.apache.zookeeper.server.quorum.QuorumPeer.recreateSocketAddresses(QuorumPeer.java:774) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:701) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:24,903] WARN Cannot open channel to 2 at election address pzoo-1.pzoo:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager) java.net.UnknownHostException: pzoo-1.pzoo at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) at java.base/java.net.Socket.connect(Socket.java:609) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:650) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:707) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectAll(QuorumCnxManager.java:735) at org.apache.zookeeper.server.quorum.FastLeaderElection.lookForLeader(FastLeaderElection.java:910) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:1247) [2019-12-30 08:27:24,903] INFO Notification time out: 12800 (org.apache.zookeeper.server.quorum.FastLeaderElection)

[solsson]

Apologies for not marking #297 as WIP. I've changed the issue title now. I only built the images and got started reading about Mirrormaker 2 and the new client.rack property. Didn't actually get to running the thing :) Contributions are welcome.

[solsson]

@jgottfriedMSC Can you verify if the above fix solves your issue?