Not works on Docker Mac Native - Zookeeper timeouts

[abtris]

I'm using Docker Native with 6GB RAM, 2CPU Core.

$ docker -v
Docker version 1.12.0, build 8eab29e

• kafka_2.11-0.10.0.0, zookeeper-3.4.6

I try make steps in help for testing.

$ docker-compose up -d
$ docker-compose scale kafka=2
$ ./start-kafka-shell.sh 172.18.0.2 172.18.0.3:2181
bash-4.3# $KAFKA_HOME/bin/kafka-topics.sh --create --topic topic \
> --partitions 4 --zookeeper $ZK --replication-factor 2
Exception in thread "main" org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server within timeout: 30000
    at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1232)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:156)
    at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:130)
    at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:75)
    at kafka.utils.ZkUtils$.apply(ZkUtils.scala:57)
    at kafka.admin.TopicCommand$.main(TopicCommand.scala:54)
    at kafka.admin.TopicCommand.main(TopicCommand.scala)

Any idea why this not work?
It's some simple test for zookeeper how check if works correctly?
IP Addresses are from docker inspect

[jonbuffington]

Most likely the broker's advertised IP address is incorrect. I use an environment variable, DOCKER_HOST_IP, to use my Mac's public IP address as the KAFKA_ADVERTISED_HOST_NAME value. Here are the relevant snippets from my scripts:

…

#
# Set the docker-compose.yml variable
#
if [[ -z "${DOCKER_HOST_IP-}" ]]; then
  docker_host_ip=$(docker run --rm --net host alpine ip address show eth0 | awk '$1=="inet" {print $2}' | cut -f1 -d'/')
  # Work around Docker for Mac 1.12.0-rc2-beta16 (build: 9493)
  if [[ $docker_host_ip = '192.168.65.2' ]]; then
    docker_host_ip=$(/sbin/ifconfig | grep -v '127.0.0.1' | awk '$1=="inet" {print $2}' | cut -f1 -d'/' | head -n 1)
  fi
  export DOCKER_HOST_IP=$docker_host_ip
fi

echo '==> building environment'

docker-compose build --pull

echo '==> launching environment'

docker-compose up -d

…

…

  kafka:
    image: wurstmeister/kafka:0.9.0.1
    links:
      - zookeeper:zk
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "${DOCKER_HOST_IP}"
      KAFKA_ZOOKEEPER_CONNECT: zk
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

…

[VatslavDS]

@jonbuffington where do you place your .bash code?

Thank you.

[jonbuffington]

@VatslavDS I created a script, bootstrap.sh, that executes the above then then docker-compose up -d. The script is in the same directory as the docker-compose.yml file although the script could be located anywhere including .bash_profile. The key part is to export DOCKER_HOST_IP to the shell that executes docker-compose.

[VatslavDS]

Can you share bootstrap.sh file please?

[sanear]

@jonbuffington Your script is truncated in some weird way, so the awk command terminates too early. If you'd upload, that would be pretty cool.

[jonbuffington]

@VatslavDS @sanear I updated the above comment with abbreviated content of the script.

[VatslavDS]

@jonbuffington I've already updated my boot.sh to your code, however when I try to publish and suscribe to that, it doesn't work (Even nor an error). Could you share your full docker-compose.yml ?

Thank you.

[mmullis]

Hi all. I had some problems switching to the new mac native docker and found that --net=host is not working as I expected.
Dropping the --net=host and individually mapping the exposed ports with -p works perfectly.
Do it for both zk and kafka.
e.g.
-p 9092:9092 for kafka

For Zookeeper, there's 3
-p 2181:2181
-p 2888:2888
-p 3888:3888 \

I hope that's on point for this problem and saves everyone some grief.

[tscholak]

@jonbuffington @VatslavDS @mmullis

I find that nothing proposed here works.

My full docker-compose.yml:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
      - "2888:2888"
      - "3888:3888"
  kafka:
    image: wurstmeister/kafka
    links:
      - zookeeper:zk
    ports:
      - "9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "${DOCKER_HOST_IP}"
      KAFKA_ZOOKEEPER_CONNECT: zk
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

What sense does it make to set KAFKA_ZOOKEEPER_CONNECT to a link alias, i.e. zk?

[sbgreene1307]

To fix this issue you can use the docker.for.mac.localhost as mentioned above. Then in your start-kafka-shell.sh add a --net=kafkadocker_default option to the docker command. Now you can use the service names 'kafka' and 'zookeeper' instead of the IP addresses in the script arguments. See the PR #274

[sscaling]

It looks like user error from the original question

$ docker-compose up -d
$ docker-compose scale kafka=2
$ ./start-kafka-shell.sh 172.18.0.2 172.18.0.3:2181

This looks like trying to use the start-kafka-shell.sh script with the internal container IPs, not the host IP.

What i'd actually expect this command to look like is

$ ./start-kafka-shell.sh 10.0.0.99 10.0.0.99:2181

where 10.0.0.99 is the public addressable IP of your machine. This is because start-kafka-shell.sh script starts a separate docker container, not on the running clusters subnet. i.e. the DNS and IPs are not routable.

the IP addresses used need to be routable from the new container.

I'll add this information to the tutorial.

[nWidart]

@sbgreene1307 I've tried your PR and instructions, however, I'm getting the following error when running the shell scripts:

docker: Error response from daemon: network kafkadocker_default not found.

Any clues as to what might be wrong? Thanks!

[sbgreene1307]

@nWidart It looks like the wurstmeister/kafka-docker changed the docker network name to kafka-docker_default (note the dash in kafka-docker). So the fork README is out of date.

But you should keep in mind that my PR was rejected. Yes it is a workaround, but should be avoided for any production usage.

[nWidart]

Thank you for your quick response!

It seems like I'm getting the same issue with the dash: docker: Error response from daemon: network kafka-docker_default not found..

I can't seem to find any reference to kafka-docker_default in the repository.

Indeed I saw the responses in the pull request. This is just for local development as an experiment, so nothing production worthy 😄

[sscaling]

@nWidart - have you read the connectivity guide. I think if you're having a specific issue with the image it would be best to open a new issue with the steps you have taken, commands you have used, versions of docker / docker-compose / host os etc.

As the issue is quite old, and only kept open for documentation, i'll open a new ticket to track the documentation issue and close this one.

[sscaling]

subsumed by #407