1

I have a Docker Compose based Swarm with a Zookeeper service that starts fine, but the Kafka service cannot start.

It keeps using the default KAFKA_ADVERTISED_LISTENERS property:

debezium_kafka.1.5hhrqfp5kqts@stephane-pc    | Using ZOOKEEPER_CONNECT=zookeeper:2181
debezium_kafka.1.5hhrqfp5kqts@stephane-pc    | Using KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://10.0.83.239:9092

And of course it cannot find the listeners specified in the map:

debezium_kafka.1.5hhrqfp5kqts@stephane-pc    | java.lang.IllegalArgumentException: requirement failed: inter.broker.listener.name must be a listener name defined in advertised.listeners. The valid options based on currently configured listeners are PLAINTEXT

My docker-compose-dev.yml file contains:

  kafka:
    image: debezium/kafka:1.2
    deploy:
      mode: global
    ports:
      - "9094:9094"
      - "9095:9095"
    networks:
      common:
    volumes:
      - "~/dev/docker/projects/debezium/volumes/kafka/data:/kafka/data"
      - "~/dev/docker/projects/debezium/volumes/kafka/logs:/kafka/logs"
    environment:
      ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: FROM_DOCKER_NETWORK://kafka:9092,FROM_HOST://kafka:9094
      KAFKA_ADVERTISED_LISTENERS: FROM_DOCKER_NETWORK://kafka:9092,FROM_HOST://localhost:9094
      KAFKA_ALLOW_PLAINTEXT_LISTENER: "yes"
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,FROM_DOCKER_NETWORK:PLAINTEXT,FROM_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: FROM_DOCKER_NETWORK
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
Stephane
  • 227
  • 3
  • 12

2 Answers2

0

I was doing something wrong, by specifying an hard coded hostname in the KAFKA_LISTENERS property:

KAFKA_LISTENERS: FROM_DOCKER_NETWORK://kafka:9092,FROM_HOST://kafka:9094

After replacing the kafka hostname by the 0.0.0.0 IP address, as in:

KAFKA_LISTENERS: FROM_DOCKER_NETWORK://0.0.0.0:9092,FROM_HOST://0.0.0.0:9094

the issue was gone and the Kafka service could start.

Note that the advertised listeners keep the container name for accessing the Kafka service from another container, and the localhost name for accessing the Kafka service from the Docker host:

KAFKA_ADVERTISED_LISTENERS: FROM_DOCKER_NETWORK://kafka:9092,FROM_HOST://localhost:9094

As explained in this tutorial the 0.0.0.0 value means the listener is listening on all interfaces. And this tutorial says subtly the same thing.

Stephane
  • 227
  • 3
  • 12
0

@stephane Thanks a lot.

I could not use kafka as the name, but had to bind it to the IP address (from docker network) and it worked.

Anand S
  • 1