Zookeeper Report Error:KeeperErrorCode = NoNode

[yijia2413]

I have read https://github.com/wurstmeister/kafka-docker/wiki/Connectivity very clearly and use the docker -compose up -d，containers look fun when startup.

But when I use kafkacat to produce messages to topic, the topic will be create automaticlly in kafka containers(it is right), but the zookeeper containers reports: Zookeeper Report Error:KeeperErrorCode = NoNode, and the kafkacat/or producer program hangup and report timeout...

here is the logs:

zookeeper_1_c0f4fc87c128 | 2018-11-21 02:37:01,839 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x167341fde5c0000 type:setData cxid:0x68 zxid:0x28 txntype:-1 reqpath:n/a Error Path:/config/topics/test1 Error:KeeperErrorCode = NoNode for /config/topics/test1
kafka_1_1b29e03b14cd | [2018-11-21 02:37:01,846] INFO Topic creation Map(test1-0 -> ArrayBuffer(1001)) (kafka.zk.AdminZkClient)

Kafka: 2.0.1
Zookeeper: 3.4.9

Is there something wrong with zookeeper network ?

[sscaling]

I'd suggest posting your config files, commands used to run the containers and commands used to run kafkacat along with any other relevant information such as Host OS / versions etc. Basically anything to reproduce the issue.

Looking at the error alone, it looks like zookeeper has been asked to set the data on node /config/topics/test1 which doesn't exist. This may indicate the broker isn't registered but a topic is trying to be created.

[yijia2413]

@sscaling Thanks, followed are some configs

HOST OS

CentOS Linux release 7.5.1804 (Core)

docker-compose.yml

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: .
    ports:
      - "9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 172.16.xxx.xxx
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "Topic1:1:3,Topic2:1:1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

where 172.16.xxx.xxx is the HOST IP

Git Commit to Use

84b3e08 update to newest kafka version 2.1.0 (#428)

Docker Version

docker version
Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:48:22 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:19:08 2018
  OS/Arch:          linux/amd64
  Experimental:     false

Start Command

docker-compose up -d
docker-compose scale kafka=3

and containers list below:
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                NAMES
fe8abc3f50ff        kafka-docker_kafka       "start-kafka.sh"         2 minutes ago       Up 2 minutes        0.0.0.0:32770->9092/tcp                              kafka-docker_kafka_3_b73ca5dcfa07
6b256a681569        kafka-docker_kafka       "start-kafka.sh"         2 minutes ago       Up 2 minutes        0.0.0.0:32769->9092/tcp                              kafka-docker_kafka_2_4faea85182e5
5d0fc530bcd2        kafka-docker_kafka       "start-kafka.sh"         2 minutes ago       Up 2 minutes        0.0.0.0:32768->9092/tcp                              kafka-docker_kafka_1_6c73b758f0f1
47cb914b8f08        wurstmeister/zookeeper   "/bin/sh -c '/usr/sb…"   2 minutes ago       Up 2 minutes        22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   kafka-docker_zookeeper_1_956aee10f68e

KafkaCat Command

echo "helloworld" | kafkacat -b 172.16.xx.xxx:32770,172.16.xx.xxx:32769,172.16.xx.xxx:32768 -P -t Topic5

(if I use Topic1 that I created at startup, the kafkacat will hang on all the time....and no logs report)

LOGS in docker

[2018-11-26 02:01:03,391] INFO Topic creation Map(Topic5-0 -> ArrayBuffer(1003)) (kafka.zk.AdminZkClient)
[2018-11-26 02:01:03,403] INFO [KafkaApi-1003] Auto creation of topic Topic5 with 1 partitions and replication factor 1 is successful (kafka.server.KafkaApis)

2018-11-26 02:01:03,384 [myid:] - INFO  [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x1674db82d7b0002 type:setData cxid:0x61 zxid:0x59 txntype:-1 reqpath:n/a Error Path:/config/topics/Topic5 Error:KeeperErrorCode = NoNode for /config/topics/Topic5

[sscaling]

Thanks for the detailed info.

For reference, here are some related issues:

• Topic creator is too brittle during new cluster construction #399
• Change how create-topics.sh checks for running Kafka #160

I can't say I honestly ever use the auto-create topics in a production, clustered environment. This is generally managed out-of-band as I want full control so I can configure access + security. This feature is more intended for allowing quick testing - but as you can see by the above two issues - it's not perfect / 100% reliable.

Looking at your copy & pasted docker ps output, it looks like all three Kafka containers were started at the same time. However, you can see the exact creation time with something like this:

$ docker ps --format 'table {{ .Names }}\t{{ .CreatedAt }}'
NAMES                                  CREATED AT
kafka-docker_kafka_2                   2018-11-26 09:28:20 +0000 GMT
kafka-docker_kafka_3                   2018-11-26 09:28:20 +0000 GMT
kafka-docker_zookeeper_1               2018-11-26 09:28:10 +0000 GMT
kafka-docker_kafka_1                   2018-11-26 09:28:10 +0000 GMT

Here i can see kafka_1 and zookeeper_1 were started 10 seconds before the other two brokers.

When I look at the logs for kafka_1, I can see that 'Topic1' creation failed because the requested replication factor was larger than the number of available brokers:

creating topics: Topic1:1:3
creating topics: Topic2:1:1
Error while executing topic command : Replication factor: 3 larger than available brokers: 1.
[2018-11-26 09:28:23,883] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.
 (kafka.admin.TopicCommand$)
Created topic "Topic2".

However inspection of my kafka_2 logs shows that "Topic1" was created successfully.

I can confirm that by describing the topics, using the tooling that comes with Kafka (exec into one of the containers then run...)

bash-4.4# kafka-topics.sh --describe --topic Topic1 --zookeeper zookeeper
Topic:Topic1	PartitionCount:1	ReplicationFactor:3	Configs:
	Topic: Topic1	Partition: 0	Leader: 1003	Replicas: 1003,1001,1002	Isr: 1003,1001,1002
bash-4.4# kafka-topics.sh --describe --topic Topic2 --zookeeper zookeeper
Topic:Topic2	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: Topic2	Partition: 0	Leader: 1001	Replicas: 1001	Isr: 1001

Checking the ports with:

$ docker ps --format 'table {{ .Names }}\t{{ .Ports }}'
NAMES                                  PORTS
kafka-docker_kafka_2                   0.0.0.0:32781->9092/tcp
kafka-docker_kafka_3                   0.0.0.0:32780->9092/tcp
kafka-docker_zookeeper_1               22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp
kafka-docker_kafka_1                   0.0.0.0:32779->9092/tcp

I can use the ports to connect using Kafkacat

$ IP=$(sudo ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1')
$ echo $IP
192.168.0.3
$ echo "foo" | kafkacat -b $IP:32781,$IP:32780,$IP:32779 -P -t Topic1
$ kafkacat -b $IP:32781,$IP:32780,$IP:32779 -C -t Topic1
foo
% Reached end of topic Topic1 [0] at offset 1

So it looks like your configuration is correct (assuming your host IP 172.16.x.x is correct).

A few things to try:

• Check each individual Kafka log, make sure at some point you see the topic is actually created (docker-compose logs kafka | grep Created)
• Use the kafka-topics.sh script to validate that the topics are correct, with a valid leader etc. Judging by your description of 'kafkacat hangs' - it's potentially created the topics, but there is no leader or ISRs do not satisfy the requirements to read/write.
• Use the kafka-topics.sh script in --create mode to see if you can create Topic1 with the required parameters.
• nuke everything and start again (to remove any state). docker-compose rm -vfs. If you have been starting / stopping containers there is likely bad state in zookeeper if you haven't torn down the environment correctly.
• Check that all your brokers are connected to zookeeper, use telnet/netcat to send dump status check to the zookeeper port. You should have consecutive broker ID's (1001,1002,1003).

$ telnet localhost 2181
Trying ::1...
Connected to localhost.
Escape character is '^]'.
dump
SessionTracker dump:
Session Sets (4):
0 expire at Mon Nov 26 09:57:44 UTC 2018:
0 expire at Mon Nov 26 09:57:46 UTC 2018:
0 expire at Mon Nov 26 09:57:48 UTC 2018:
3 expire at Mon Nov 26 09:57:50 UTC 2018:
	0x1674f59099c0004
	0x1674f59099c0000
	0x1674f59099c0003
ephemeral nodes dump:
Sessions with Ephemerals (3):
0x1674f59099c0000:
	/brokers/ids/1001
	/controller
0x1674f59099c0003:
	/brokers/ids/1002
0x1674f59099c0004:
	/brokers/ids/1003

[yijia2413]

@sscaling Thanks for the answer, I learnt a lot.

I solved the problem by adding the following to /etc/hosts

172.27.0.5  a5f54c045126
172.27.0.4  d64beffb4b37
172.27.0.3  c016ef53f218

which is the container-host and container-ip mapping.

issue closed.

[psdon]

any explanation why I need to manually put those to /etc/hosts?

[juandisay]

any explanation why I need to manually put those to /etc/hosts?

agree, any explanation?