Unable to create Kafka topics with Kafka and Zookeeper running on Docker

Question

I have Kafka and Zookeeper running on two separate Docker containers:

<private-domain>/wurstmeister-kafka:0.10.1.0-2
<private-domain>/wurstmeister-zookeeper:3.4.9

Both containers seem to be up, but when I try to create Kafka topics by getting in to the first container:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

I get this error:

java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2020-06-07 03:10:55,293] WARN Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)

Please notice that I did read other related questions and tried adding arguments to the command, such as -e ZK_HOSTS="localhost:2181". I know of other people working in the environment as mine who were able to run the commands successfully, so I suspect this might be a configuration issue on my side. Can you please guide?

EDIT: Here are the Docker Compose files:

version: '2'
services:
 kafka:
   image: <private-domain>/wurstmeister-kafka:0.10.1.0-2
   container_name: kafka
   ports:
      - 9092:9092
   environment:
     KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
     KAFKA_ADVERTISED_PORT: 9092
     KAFKA_ZOOKEEPER_CONNECT: 127.0.0.1:2181

   restart:
       "unless-stopped"

and

version: '2'
services:
 zk:
  image: <private-domain>/wurstmeister-zookeeper:3.4.9
  container_name: zk
  ports:
      - "2181:2181"
  restart:
   "unless-stopped"

and the output of docker ps:

CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                                                NAMES
bf67a49da57a        wurstmeister-kafka:0.10.1.0-2   "start-kafka.sh"         5 months ago        Up 29 minutes       0.0.0.0:9092->9092/tcp                               kafka
ef3e908d82b3        wurstmeister-zookeeper:3.4.9    "/bin/sh -c '/usr/sbin/sshd && bash /usr/bin/start-zk.sh'"   5 months ago        Up 29 minutes       22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   zk

@RohitYadav yes but can you help more based on the information I have provided? What should I do? — BlameMe, Jun 07 '20 at 04:02

score 1 · Answer 1 · answered Jun 09 '20 at 03:58

You have two Compose files. Thus, Your containers are on separated networks, and cannot refer each other.

You must add both services in one file, under one services: block, and run only one docker-compose up command

You can find working compose files here across the internet, or you could use minikube / oc with Kafka Helm Charts or Operators, which is how the large companies are testing Kafka in containers.

Unable to create Kafka topics with Kafka and Zookeeper running on Docker

1 Answers1