Unable to connect to Kafka from outside application

Question

I have two docker machines and I want to create a kafka cluster inside docker swarm. My docker-compose.yml looks like this:

version: '3.2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - "9092:9092"
      - "29092:29092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_LISTENERS: PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092

I followed this question: Unable to connect to Kafka run in container from Spring Boot app run outside container and I am trying to access kafka from outside using localhost:29092.

I have already create the topic mytesttopic inside kafka. The below python code:

from kafka import KafkaConsumer, SimpleProducer, TopicPartition, KafkaClient


def consume_from_topic():
    try:
        consumer = KafkaConsumer('mytesttopic',
                                 group_id= None,
                                 bootstrap_servers=['localhost:29092'],
                                 auto_offset_reset='earliest')

        for message in consumer:
            #consumer.commit()
            print ("%s:%d:%d: key=%s value=%s" % (message.topic, message.partition,
                                                  message.offset, message.key,
                                                  message.value))
    except Exception as e:
        print(e)
        pass



if __name__ == '__main__':
    consume_from_topic()

returns:

NoBrokersAvailable

Does anyone know what I am missing here?

You mentioned you are running docker swarm over two machines, from which source are you running the python application? One of the docker nodes or your local machine? — Sven Hakvoort, Feb 18 '20 at 12:41

score 0 · Answer 1 · answered Feb 18 '20 at 12:54

0

Given you are running docker swarm on 2 other machines you won't be able to connect on localhost:29092 due to the fact that kafka will be exposed on port 29092 on your nodes of the docker swarm. Try connecting to kafka by using the hostname of one of your nodes + port 29092. You should be able to connect to kafka this way.

Please note that this will only work if you are running docker swarm with routing mesh, the routing mesh makes sure that each node accepts incoming requests on a published port for any service, no matter if it is running on the same hosts and makes sure the traffic reaches the actual host where your container is running.

If you have not yet setup routing mesh try connceting to the actual hostname where a kafka container is running (not recommended, but for testing purposes it works)

I hope this helps you!

answered Feb 18 '20 at 12:54

Sven Hakvoort

3,543
2
17
34

the hostname is manager. you mean bootstrap_servers=['manager:29092'] ? If so, this does not work – e7lT2P Feb 18 '20 at 13:50
@Telperinquar, can you ping manager:29092? and this is one of your docker nodes which you can reach with this hostname? – Sven Hakvoort Feb 18 '20 at 13:57
No I cannot ping. I found the hostname with: docker node ls, right? – e7lT2P Feb 18 '20 at 13:59
@Telperinquar, that is the hostname of that node within the swarm itself, you need the IP address of this node or the hostname which you would also for SSH to your node – Sven Hakvoort Feb 18 '20 at 14:00
the ip is 192.168.99.100 and I cannot ping – e7lT2P Feb 18 '20 at 14:06
the same. I do not understand what I am doing wrong. – e7lT2P Feb 18 '20 at 14:12
One question, I can only solve the problem with routing mesh? Lets suppose that i have a swarm and I want to connect not via the hostname. Is there exist a solution? – e7lT2P Feb 18 '20 at 14:18
@Telperinquar, no it's not possible to connect without a hostname, however a suitable approach would be to use a service like traefik with a constraint on the manager node, you would then setup a DNS record for your master node which will be the way to connect to your swarm, within traefik you would then configure the subdomains and port of each respective service. You would then be able to connect to kafka for example in a way like this: kafka.yourdomain.com – Sven Hakvoort Feb 18 '20 at 14:32
@Telperinquar, also are you on the same network as your docker machines? Or are they located elsewhere? And to be sure a telnet on 192.168.99.100 port 29092 does not work? And have you tried setting bootstrap_servers=['192.168.99.100:29092'] – Sven Hakvoort Feb 18 '20 at 14:41

score 0 · Answer 2 · answered Feb 18 '20 at 16:10

0

Your listeners are the exact same.

You need to set PLAINTEXT_HOST://0.0.0.0:29092 to bind the listener to all interfaces

answered Feb 18 '20 at 16:10

OneCricketeer

179,855
19
132
245

Unable to connect to Kafka from outside application

2 Answers2