Cannot connect to kafka docker container from outside client (wurstmeister images)

Question

There are so many answers for this question that I ended up being totally confused about how I can connect to Kafka docker container from an outside client.

I have created two docker machines, a manager and a worker with these commands:

docker-machine create manager
docker-machine create worker1

I have add these two nodes inside a docker swarm.

docker@manager:~$ docker node ls                                                                                                                                                                             
ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
6bmovp3hr0j2w5irmexvvjgzq *   manager             Ready               Active              Leader              19.03.5
mtgbd9bg8d6q0lk9ycw10bxos     worker1             Ready               Active                                  19.03.5

docker-compose.yml:

version: '3.2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - target: 9094
        published: 9094
        protocol: tcp
        mode: host
    environment:
      HOSTNAME_COMMAND: "hostname | awk -F'-' '{print $$2}'"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

From inside docker, everything works fine. I can create topics and then produce/consume messages.

I created a python script in order to consume messages from outside docker. The simple code is presented below:

from kafka import KafkaConsumer
import json

try:
    print('Welcome to parse engine')
    consumer = KafkaConsumer('streams-plaintext-input', bootstrap_servers='manager:9094')
    for message in consumer:
        print(message)
except Exception as e:
    print(e)
    # Logs the error appropriately. 
    pass

But the code is stack forever. The connection is not correct. Can anyone provide any help on how to setup a connection?

Answer 1

Since you are using docker-machine you have to either

1. Run your code also in a container (using kafka:9092)
2. Run your code within the VM OS (using vm-host-name:9094)
3. Add PLAINTEXT://localhost:9096 to the advertised listeners, expose 9096 from the VM to your host, then use localhost:9096 in your code (note: 9096 is some random port)

---

The gist is that clients must be able to connect to the bootstrap address and the advertised one that is being returned. If it cannot connect to the second, code will timeout.