From 4a49ff3bf11d1eea74735d65d5477d06f2ff3998 Mon Sep 17 00:00:00 2001 From: Staffan Olsson Date: Sun, 8 Mar 2020 17:02:46 +0100 Subject: [PATCH] Seeing occasinal connection on zookeeper on GKE e2-small instances. and I'm guessing it could be due to throttling Kafka for example reports: kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING And zookeeper logs include things like: java.io.EOFException at java.base/java.io.DataInputStream.readInt(DataInputStream.java:397) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206) [2020-03-06 12:35:45,056] WARN Connection broken for id 1, my id = 4, error = (org.apache.zookeeper.server.quorum.QuorumCnxManager) java.net.SocketException: Socket closed at java.base/java.net.SocketInputStream.socketRead0(Native Method) at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168) at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:271) at java.base/java.io.DataInputStream.readInt(DataInputStream.java:392) at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1206) I've been able to resolve it twice with this memory limit increase, but of course it could also be that the restart resolves the issue. --- zookeeper/50pzoo.yml | 2 +- zookeeper/51zoo.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/zookeeper/50pzoo.yml b/zookeeper/50pzoo.yml index 92b03983..9bc0d012 100644 --- a/zookeeper/50pzoo.yml +++ b/zookeeper/50pzoo.yml @@ -63,7 +63,7 @@ spec: name: leader-election resources: requests: - cpu: 10m + cpu: 30m memory: 100Mi limits: memory: 120Mi diff --git a/zookeeper/51zoo.yml b/zookeeper/51zoo.yml index adcc6e24..d50e9708 100644 --- a/zookeeper/51zoo.yml +++ b/zookeeper/51zoo.yml @@ -65,7 +65,7 @@ spec: name: leader-election resources: requests: - cpu: 10m + cpu: 30m memory: 100Mi limits: memory: 120Mi