How to use confluent/cp-kafka image in docker compose with advertising on localhost and my network container name kafka?

Question

How to use confluent/cp-kafka image in docker compose with exposing on localhost and my network container name kafka?

Do not link this as duplicate of:

1. Connect to docker kafka container from localhost and another docker container
2. Cannot produce message to kafka from service running in docker

These do not solve my issue because the methods they use are depreciated by confluent/cp-kafka and I want to connect on localhost and on the docker network.

In the configure script on confluent/cp-kafka they do this annoying task:

# By default, LISTENERS is derived from ADVERTISED_LISTENERS by replacing
# hosts with 0.0.0.0. This is good default as it ensures that the broker
# process listens on all ports.
if [[ -z "${KAFKA_LISTENERS-}" ]]
then
  export KAFKA_LISTENERS
  KAFKA_LISTENERS=$(cub listeners "$KAFKA_ADVERTISED_LISTENERS")
fi

It always sets whatever I give KAFKA_ADVERTISED_LISTENERS to 0.0.0.0! Using the docker network, doing

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9093,PLAINTEXT://kafka:9093

I expect the listeners to be either localhost:9092 or 0.0.0.0:9092 and some docker ip PLAINTEXT://172.17.0.1:9093 (whatever kafka resolves to on the docker network)

Currently I can get only one or the other to work. So using localhost, it only works on the host system, no docker containers can access it. Using kafka, it only works in the docker network, no host applications can access it. I want it to work with both. I am using docker compose so that I can have zookeeper, kafka, redis, and my application start up. I have other applications that will startup without docker.

Update So when I set PLAINTEXT://localhost:9092 I can access kafka running docker, outside of docker.

When I set PLAINTEXT://kafka:9092 I cannot access kafka running docker, outside of docker.

This is expected, however doing this: PLAINTEXT://localhost:9092,PLAINTEXT://kafka:9093 I would expect to access kafka running docker, both inside and outside docker. The confluent/cp-kafka image is wiping out localhost and kafka. Setting them both to 0.0.0.0, then throwing an error that I set 2 different ports to the same ip...

Maybe I'm just clashing into some opinionated docker image and should look for a different image...

Answer 1

Maybe I'm just clashing into some opinionated docker image and should look for a different image...

The image is fine. You might want to read this explanation of the listeners.

tl;dr - you don't want to (and shouldn't?) use the same listener "protocol" in different networks.

Use the advertised.listeners, no need to edit the listeners

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:9092,PLAINTEXT_HOST://localhost:29092

---

When PLAINTEXT://localhost:9093 is being loaded inside of the container, you need to add port mappings for 9093, which should be self explanatory, and you connect to localhost:9093 and it should work.

Then, if you also had PLAINTEXT://kafka:9092, that will only work within the Docker Compose network overlay, not externally to your DNS servers, because that's how Docker networking works. You should be able to run other applications as part of that Docker network with the --network flag, or link containers using Docker Compose

Keep in mind that if you're running on Mac, the recommended way (as per the Confluent docs) is to run these containers in Docker Machine, in a VM, where you can manage the external port mappings correctly using the --net=host flag of Docker. However, using the blog above, it all works fine on a Mac outside a VM.