Kafka communication inside docker network fails, outside of docker works

Question

I have setup a kafka-zookeeper compose in docker with three listeners:

1. INTERNAL on Port 9092
2. EXTERNAL_SAME_HOST on 29092
3. EXTERNAL_DIFFERENT_HOST on 29093

Server with docker has the IP 192.168.66.66. I tested it, and 2 and 3 are reachable via my kafka-python test scripts. 1 however (Tested from inside a container) fails throwing "No brokers available". What did I do wrong with INTERNAL?

My docker-compose.yml:

version: "3.8"
services:
  #########
  # Kafka #
  #########

  zookeeper:
    container_name: zookeeper
    image: wurstmeister/zookeeper

    networks:
      - kafka_network

    ports:
      - "2181:2181"

  kafka:
    container_name: kafka
    image: wurstmeister/kafka

    networks:
      - kafka_network

    ports:
      - "29092:29092"
      - "29093:29093"

    expose:
      - "9092"

    environment:
      # Using three ways to reach kafka: From INSIDE docker (Other containers), from outside docker but running on the same server as docker (EXTERNAL_SAME_HOST), and from another computer.
      KAFKA_LISTENERS: EXTERNAL_SAME_HOST://:29092,EXTERNAL_DIFFERENT_HOST://:29093,INTERNAL://:9092
      
      # Publishing the above ports
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL_SAME_HOST://localhost:29092,EXTERNAL_DIFFERENT_HOST://192.168.66.66:29093
      
      # Setting the security protocol for all three listeners to PLAINTEXT
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT,EXTERNAL_DIFFERENT_HOST:PLAINTEXT
      
      # Settings for zookeeper-kafka-communication
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      
      # Create Kafka topics "NAME:PARTITION:REPLICAS:RETENTION_POLICY,..."
      KAFKA_CREATE_TOPICS: "debug:1:1:delete,debug2:1:1:delete" # Some test topics for debugging purposes
      
      # To encourage correct API use, forbid automatic topic creation (Only topics created by an explicit command or the topic creation above can be used. Prevents typos)
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

    depends_on:
      - zookeeper

    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

networks:
  kafka_network:
    name: kafka
    driver: bridge

My kafka-python test producer.py:

from kafka import KafkaProducer
import json
import time

kafka_ip = 'kafka:9092'
topic = 'debug'

producer = KafkaProducer(bootstrap_servers=[kafka_ip],
                         value_serializer=lambda x:
                         json.dumps(x).encode('utf-8'))

while True:
    data = 'INTERNAL listener test'
    producer.send(topic, value=data)
    time.sleep(1)

And the dockerfile I used for the producer:

FROM python:3.8-alpine
COPY requirements.txt /
RUN pip install -r requirements.txt
COPY src/ /
CMD [ "python", "./producer.py" ]

I created the container via

docker build --no-cache -t kafka_internal_test_producer .

and ran it with

docker run -d --name kafka_internal_test_producer kafka_inter
nal_test_producer

Answer 1

See the comments under the question, the producer was running on the wrong network. Setting it to "kafka" via docker connect/portainer UI worked. Thanks @Robin Moffatt for the quick solution!