Can't initialize transactions

Question

I have local instances of Kafka and Zookeeper running in Docker on my local machine (macOS Monterey) which have worked quite well for my needs so far. I recently want to implement something with Kafka transactions and run into the problem that I always get an error

(KafkaError{code=COORDINATOR_NOT_AVAILABLE,val=15,str="Failed to initialize Producer ID: Broker: Coordinator not available"}).

Several restarts with and without deleting the data did not help either.

This is a minimal, reproducible example in Python:

#!/usr/bin/python3

import confluent_kafka
import sys

topic = "test-topic"
config = {
    "bootstrap.servers": "localhost:9092",
    "enable.idempotence": True,
    "transactional.id": "test-0",
}

producer = confluent_kafka.Producer(config)
producer.init_transactions(300.0)

And I'm using this docker-compose.yaml file to create the Kafka and the Zookeeper server:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181
    volumes:
      - ./data/zookeeper/data:/var/lib/zookeeper/data
      - ./data/zookeeper/log:/var/lib/zookeeper/log

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
      - 29092:29092
    hostname: kafka
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CONFLUENT_BALANCER_TOPIC_REPLICATION_FACTOR: 1
    volumes:
      - ./data/kafka:/var/lib/kafka/data

With a shorter timeout of 2.0 seconds I get the following output:

clemens@morticia kafka % python3.9 transaction.py
%4|1661869211.606|GETPID|rdkafka#producer-1| [thrd:main]: Failed to acquire transactional PID from broker TxnCoordinator/1: Broker: Coordinator not available: retrying
%4|1661869212.115|GETPID|rdkafka#producer-1| [thrd:main]: Failed to acquire transactional PID from broker TxnCoordinator/1: Broker: Coordinator not available: retrying
%4|1661869212.629|GETPID|rdkafka#producer-1| [thrd:main]: Failed to acquire transactional PID from broker TxnCoordinator/1: Broker: Coordinator not available: retrying
%4|1661869213.141|GETPID|rdkafka#producer-1| [thrd:main]: Failed to acquire transactional PID from broker TxnCoordinator/1: Broker: Coordinator not available: retrying
Traceback (most recent call last):
  File "/Users/clemens/docker/kafka/transaction.py", line 14, in <module>
producer.init_transactions(2.0)
cimpl.KafkaException: KafkaError{code=COORDINATOR_NOT_AVAILABLE,val=15,str="Failed to initialize Producer ID: Broker: Coordinator not available"}

Workaround:

I “solved” this issue by using another Docker image (confluentinc/cp-server instead of confluentinc/cp-kafka), and I use now a compose file, which also starts the schema registry. Where I used this compose file as a base.

However, I have not checked exactly what the use of transactions allows, but I remain highly interested in what is causing the problem.