KAFKA_CFG_LISTENERS must include a listener for CONTROLLER

Question

Edit:

For anyone reading this (question maintained below) here is some short summary which might resolve your issue:

• The Bitnami Kafka 3.4 image appears to have been updated recently to use Kraft instead of Zookeeper by default
• To revert back to Zookeeper, add this to your environment variables in docker-compose.yml: KAFKA_ENABLE_KRAFT=no.
• See also: https://github.com/bitnami/containers/tree/main/bitnami/kafka

Essentially my problem appears to have been something like this, although I have only just come to this realization:

• My server which hosted Kafka (via Docker) power cycled
• As part of the restart process to get the docker container running again, it pulled an updated image from Bitnami
• This updated image switched to Kraft by default
• This meant my compose file was now in broken state because of the image update and the new defaults which came with it

---

I had a working Kafka development environment. It died after a reboot, and now I can't get it working again.

I cleared my system, removing all Docker containers, and attempted to start again.

My docker-compose.yml which was obtianed from https://hub.docker.com/r/bitnami/kafka/ is shown below.

There are a couple of modifications here, such as the port, and the listener config.

curl -sSL https://raw.githubusercontent.com/bitnami/containers/main/bitnami/kafka/docker-compose.yml > docker-compose.yml

$ cat docker-compose.yml
version: "2"

services:
  kafka:
    container_name: "kafka1"
    image: docker.io/bitnami/kafka:3.4
    networks:
      - "kafka_network"
    ports:
      - "29092:29092"
    volumes:
      - "kafka_data:/bitnami"
    environment:
      - KAFKA_CFG_LISTENERS=EXTERNAL_SAME_HOST://0.0.0.0:29092,INTERNAL://0.0.0.0:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=EXTERNAL_SAME_HOST://localhost:29092,INTERNAL://kafka1:9092
      - ALLOW_PLAINTEXT_LISTENER=yes

volumes:
  kafka_data:
    driver: local

networks:
  kafka_network:
    name: "kafka_network"

The docker container does not start when started with docker compose up -d.

I found this line in the log file. (/var/lib/docker/containers/3bd1.../3bd1...-json.log)

...
WARN KAFKA_CFG_LISTENERS must include a listener for CONTROLLER
...

What confuses me about that is that I have previously asked this question on Stack Overflow:

• Are the strings passed to Kafka for the configuration of the "listeners" and "advertised listeners" arbitrary?

The answer to that previous question didn't provide any information about how Kafka interprets these strings, but it also didn't disagree that they are arbitrary.

I am now very confused. I tried to start Kafka, and it failed to start producing a warning suggesting that I need to have a particular string (in this case "CONTROLLER") used as part of my environment variables. This is presumably for one, or both, of the Kafka Listeners properties.

In short - I do not know what I am doing. This might be a total red herring. Essentially, the new version of the Kafka docker configuration (docker-compose.yml) doesn't seem to be working anymore.

I notice that the new docker-compose.yml doesn't say anything about zookeeper. I don't fully understand why this is. I believe it is the case that zookeeper was intended to be depricated, or removed, as a requirement. Maybe that happened in the last couple of weeks?

Answer 1

Bitnami Kafka tag 3.4 defaults to start Kafka in Kraft mode. Specifically, image tag 3.4.0-debian-11-r22 switched this (older versions still require Zookeeper), and this was released in the last month. This means it needs a Raft controller, and not Zookeeper (KIP-500 is a document that explains why Zookeeper is being removed). That controller needs to be a defined listener. Its name is still arbitrary (see control.plane.listener.name)

There's an outgoing issue that the new mode doesn't work properly in that container because they decided to do special logic to parse the value, and somehow CONTROLLER became a required string, even though it isn't for Kafka itself.

If you had a working Compose file, then you could have kept it. Deleting your docker images doesn't mean you needed a fresh start for everything

Regarding persisting your data, refer https://github.com/bitnami/containers/tree/main/bitnami/kafka#persisting-your-data