0

Context

I am trying to mimic a distributed Kafka cluster via docker-compose projects, all are running on a single docker host. Cluster is having 2 brokers with a zookeeper, each running on its own separate docker-compose project.

docker-compose projects

  1. zookeeper
  2. broker-1
  3. broker-2

So I have created a topic called "test-topic" with 2 partitions and a replication factor of 2. (replication factor was decided based on each broker keeping a copy of each partition )

Problem

but I ran into the following warning/error when running a producer client inside any broker container.

kafka-console-producer --topic test-topic --bootstrap-server host.docker.internal:9093

WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 135 : {test-topic=INVALID_REPLICATION_FACTOR} (org.apache.kafka.clients.NetworkClient)

I can't seem to figure out why the replication factor is invalid, as there are sufficient brokers to accommodate the replication requirements.

Also running a zookeeper shell in each broker seems to give desired partition distribution for the topic, which rules out if one of the brokers is offline/ or not having the partition.

# zookeeper shell  - zookeeper-shell host.docker.internal:2181
get /brokers/topics/test-topic

{"partitions":{"0":[1,2],"1":[2,1]},"topic_id":"_zLuyBjBTkm5SSTnVjMYQA","adding_replicas":{},"removing_replicas":{},"version":3}

Kafka Configurations

  • zookeeper is mapping the 2181 port to the host and two brokers are connecting to the zookeeper via host.docker.internel:2181. (mind that docker desktop is running in Windows).
  • I have ports 9093 (broker-1), 9095 (broker-2) mappings for host machine for respective brokers, while them being the advertised listeners.

here are the two compose files for brokers.

broker 1

version: "3"
services:
  broker1:
    image: confluentinc/cp-kafka:7.3.2
    container_name: kd-broker-1
    ports:
      - "9093:9093"
      - "9094:9094"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: "host.docker.internal:2181"
      KAFKA_LISTENERS: INTERNEL://:9092, EXTERNEL://:9093, DEVELOPMENT://:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNEL:PLAINTEXT,EXTERNEL:PLAINTEXT, DEVELOPMENT:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNEL://:9092, EXTERNEL://host.docker.internal:9093, DEVELOPMENT://localhost:9094
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNEL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2

broker 2

version: "3"
services:
  broker2:
    image: confluentinc/cp-kafka:7.3.2
    container_name: kd-broker-2
    ports:
      - "9095:9095"
      - "9096:9096"
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: "host.docker.internal:2181"
      KAFKA_LISTENERS: INTERNEL://:9092, EXTERNEL://:9095, DEVELOPMENT://:9096
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNEL:PLAINTEXT,EXTERNEL:PLAINTEXT, DEVELOPMENT:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INTERNEL://:9092, EXTERNEL://host.docker.internal:9095, DEVELOPMENT://localhost:9096
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNEL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
melkorCba
  • 458
  • 5
  • 9

1 Answers1

0

No services are running on your host. You should remove host.docker.internal mappings, and not use that address to connect to the servers.

Use only one Compose file with one (or zero) Zookeepers, and 1 or more brokers. Note: Running more than one brokers on one host is not truly fault-tolerant.

If you must use more than one compose file, then use a shared Docker network across all containers.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • This is a kind of POC. My final goal is to make a deployment modal where there is one main service with the Kafka cluster ( with 3 brokers). The additional services will be hosted in an entirely different host (may it be VM, or any other host). each of these services is defined in a compose file where it has a local Kafka broker which connects to the main cluster via the internet (zookeeper IP). So having everything in the same compose file is not what I am trying to test. – melkorCba May 15 '23 at 17:21
  • Kafka cluster should be in one compose file for sure, otherwise replication will not work, as your error says. Zookeeper is no longer needed for Kafka, but you should not expose to "the internet"... You can use other files for _other services_. But you need a network bridge. https://docs.docker.com/compose/compose-file/06-networks/ ... If you want to simulate VMs, then use Vagrant or Multipaas rather than Docker Compose – OneCricketeer May 15 '23 at 21:33