0

The post here was already referenced before asking this question. I think the details of how to use different ports in the configuration is not covered there. The question at large is still the same: different ports on left and right side, then which port goes into the LISTENERS, what goes into ADVERTISED_LISTENERS and what port should be used by the external host client with bootstrap.servers?

This topic has been discussed many times on stackoverflow.com and I have kind of understood the concept. LISTENERS -> meant for initial connection so that clients can get the metadata of how to talk to the brokers. ADVERTISED_LISTENERS -> host/IP:port info to be used for the communicating with the brokers. The only thing I haven't understood is the port mapping. For example, in the below config from Docker Compose file, when I set "3080:3080" (and 3080 on the appropriate ADVERTISED_LISTENERS as well, things work as expected). My client running in Kubernetes on an external system can talk to Kafka. However, when I change the right hand side port to 3081 (and same in ADVERTISED_LISTENERS), the client errors out saying "Broker may not be available".

version: "2"

services:
  zookeeper:
    image: docker.io/bitnami/zookeeper:3.8
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: docker.io/bitnami/kafka:3.4
    ports:
      - "29092:29092"
      - "3080:3081"
    volumes:
      - "kafka_data:/bitnami"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_CFG_LISTENERS=INTERNAL_TO_DOCKER://:9092,EXTERNAL_SAME_HOST://:29092,EXTERNAL_DIFFERENT_HOST://:3081
      - KAFKA_CFG_ADVERTISED_LISTENERS=INTERNAL_TO_DOCKER://kafka:9092,EXTERNAL_SAME_HOST://localhost:29092,EXTERNAL_DIFFERENT_HOST://<DOCKER-HOST-IP>:3081
      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL_TO_DOCKER:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT,EXTERNAL_DIFFERENT_HOST:PLAINTEXT
      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INTERNAL_TO_DOCKER
    depends_on:
      - zookeeper

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

Error from client:

2023-02-28 20:20:05.242  WARN 1 --- [-StreamThread-1] org.apache.kafka.clients.NetworkClient   : [Consumer clientId=<MY-CLIENT-ID>, groupId=<MY-GROUP-ID>] Connection to node 1001 (/<DOCKER-HOST-IP>:3081) could not be established. Broker may not be available.

What is the significance of left and right side ports in port-mapping with respect to the LISTENERS and ADVERTISED_LISTENERS. Is there not a way to use different ports like "a:b"? On the client side, the BOOTSTRAP_SERVERS string I am using is: "--set global.kafka.bootstrapServers=DOCKER-HOST-IP:3080 --set global.kafka.exposedBootstrapServers=DOCKER-HOST-IP:3080"

curious_brain
  • 391
  • 2
  • 17
  • The right side of Compose ports mapping is the container port. This is **only** applicable to `KAFKA_CFG_LISTENERS`... Not really clear what your Compose file is to help debug with Kubernetes ClusterIP / NodePort resources. Perhaps you should use Strimzi instead? https://strimzi.io/blog/2019/04/17/accessing-kafka-part-1/ – OneCricketeer Feb 28 '23 at 21:54
  • `EXTERNAL_SAME_HOST` does not need to use `localhost`, btw. You can replace that with `` and only need two advertised listeners – OneCricketeer Feb 28 '23 at 21:54
  • In the reference post you have mentioned while closing this question, this is mentioned: "To verify the ports are mapped correctly on the host, ensure that docker ps shows the kafka container is mapped from 0.0.0.0: -> /tcp". With this explanation, the right side port should be used with ADVERTISED_LISTENERS. Nevertheless, if I use 3081 with LISTENERS and 3080 with ADVERTISED_LISTENERS, what should be the port used with bootstrap.servers property on the client side that is on a different host than DOCKER-HOST? – curious_brain Feb 28 '23 at 22:48
  • The left side is the host port. That's what ps output shows. From an external client, that's what needs to be advertised, yes. But also the right side needs to be in listeners because that's the actual server inside the container. Kafka is weird because they really cannot be different/re-mapped. Read the linked Confluent blog in that post. In reference to kubernetes, `docker ps` isn't what you want. You want `kubectl get services` and look at the exposed port. – OneCricketeer Mar 01 '23 at 02:57

0 Answers0