docker kafka broker is not accessible from outside or other docker container

Question

I'm running kafka broker in docker containers, zookeeper in a separate container.

I tried to reach the zookeeper from my local, It works fine.

I tried to reach the broker from my local, it throws the below error and eventually fails.

[Timestamp] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 65 : {tmpc=INVALID_REPLICATION_FACTOR} (org.apache.kafka.clients.NetworkClient)
[Timestamp] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 66 : {tmpc=INVALID_REPLICATION_FACTOR} (org.apache.kafka.clients.NetworkClient)

I used port (-p and --link) and I also tried with a separate network. I'm able to access zookeeper in both scenarios but I can't access the broker.

I have a base image which just has kafka unzipped and java installed in it. It runs in ubuntu

Docker image for zookeeper:

FROM IMAGE
COPY ./zookeeper_setup.sh  zookeeper_setup.sh
ENTRYPOINT ["sh","zookeeper_setup.sh"]

Zookeeper setup: # Just changing the log dir so I can mount it to a location and using the same 2181 port by default

sed -i "s%dataDir=/tmp/zookeeper%dataDir=${LOGDIR}%g" ./config/zookeeper.properties
sh ./bin/zookeeper-server-start.sh ./config/zookeeper.properties

Kafka image:

FROM IMAGE [SAME base image]
COPY ./broker_setup.sh  broker_setup.sh
ENTRYPOINT ["sh","broker_setup.sh"]

Broker setup # just setting up the internal and external listeners, ports and zookeeper:

sed -i "s#log.dirs=/tmp/kafka-logs#log.dirs=${LOGDIR}#g" ./config/server.properties
sed -i "s#broker.id=0#broker.id=${BROKERID}#g" ./config/server.properties
sed -i "s@#listeners=PLAINTEXT://:9092@listeners=INTERNAL://:2${KAFKAPORT},EXTERNAL://:${KAFKAPORT}@g" ./config/server.properties
sed -i "s/zookeeper.connect=localhost:2181/zookeeper.connect=${ZOOKEEPERCONNECT}/g" ./config/server.properties
echo -e "\nlistener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT" >> ./config/server.properties
echo -e "\ninter.broker.listener.name=INTERNAL" >> ./config/server.properties
sed -i "s~#advertised.listeners=PLAINTEXT://your.host.name:9092~advertised.listeners=INTERNAL://localhost:2${KAFKAPORT},EXTERNAL://localhost:${KAFKAPORT}~g" ./config/server.properties
sh ./bin/kafka-server-start.sh ./config/server.properties

I set up the brokers to expose the port to the outside world by referring to this page. https://rmoff.net/2018/08/02/kafka-listeners-explained/

But when I start the container :

docker run -d -e LOGDIR=/kafka/logs/zookeeper/ -p 2281:2181 -v [OutsideDir]:/kafka/logs/zookeeper/ --name zookeeper-client -t Image

docker run -d --link zookeeper-client:zookeeper --name broker0 -e LOGDIR=/kafka/logs/kafka-logs-1/ -e BROKERID=0 -e KAFKAPORT=9093 -e ZOOKEEPERCONNECT=zookeeper:2181 -e ALLOW_PLAINTEXT_LISTENER=yes -p 8093:9093 -p 28093:29093 -v [LogDir_2]:/kafka/logs/kafka-logs-1/ -t Image

Using binaries, I can create topics contacting the zookeeper from my local and from inside of another container. But I can't connect to broker in order to produce any message.

I tried it using an isolated docker network.

docker network create app-network

docker run -d --network app-network --network-alias app-net-zoo -e LOGDIR=/kafka/logs/zookeeper/ -p 2281:2181 -v [logDir]:/kafka/logs/zookeeper/ --name zookeeper-client -t image


docker run --network app-network --network-alias app-net-broker --name broker0 -e LOGDIR=/kafka/logs/kafka-logs-1/ -e BROKERID=0 -e KAFKAPORT=9093 -e ZOOKEEPERCONNECT=app-net-zoo:2181 -e ALLOW_PLAINTEXT_LISTENER=yes -p 8093:9093 -p 28093:29093 -v [logDir_2]:/kafka/logs/kafka-logs-1/ -t image

docker run --network app-network --network-alias app-net-it -it Image_BASE

Tried to connect with the broker from the base image container using

kafka-console-producer.sh --broker-list app-net-broker:9093 --topic tempC [Did not work]

kafka-topics.sh --create --topic Name_T --zookeeper app-net-zoo:9093 [Works fine]

Killed the broker and zookeeper, cleared the logs and then I created a zookeeper container alone, then launched the base image as container. in that Interactive container I created a broker and I tried to connect to that broker, it works fine. Now I can clearly see this is a pure network issue. Can you folks help me with this ?

Please explain why you're bothering to build your own image rather than use existing ones — OneCricketeer, Sep 04 '22 at 17:17
You still have a dependency on the Apache download mirrors, which remove versions over time. Also, your `ALLOW_PLAINTEXT_LISTENER` variable doesn't seem to be used, and that's only for the bitnami containers — OneCricketeer, Sep 06 '22 at 14:04

score 0 · Accepted Answer · answered Sep 04 '22 at 17:24

You have two advertised listeners, both of which are localhost. Therefore, no client running within the Docker network will work, only ones running on your same host using a port forward will be able to access the broker post-bootstrap.

The reason your port forward isn't working is because you've set -e KAFKAPORT=9093, but you've forwarded port 8083 on the host to 9093 in the container, meanwhile Kafka will return 9093 after bootstrapped for future connections, as that's what is being advertised. This is perfectly explained in the blog post you've linked to...

Please read through Connect to Kafka running in Docker

Exposed it to the same port 9093 and it worked. Thanks – tempUser Sep 06 '22 at 03:27 — tempUser, Sep 06 '22 at 03:27

docker kafka broker is not accessible from outside or other docker container

1 Answers1