why should i use docker image "confluentinc/kafka" to kafka cluster?

Question

I did

docker-compose up

for kafka clustering in my local.

and whenever i want to make topic, retrieve topics i have or search data stored in the topic i use

docker run --net=host --rm confluentinc/cp-kafka:latest kafka-topics --describe --topic bar --zookeeper localhost:32181

which is on the official kafka clustering deployment site.

However i really want to use Kafka in my local storage not the kafka docker image like

kafka-topics --describe --topic bar --zookeeper localhost:32181

if i use the code above i face this error.

Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:258)

Besides if i check netstat -anp tcp there is no LISTEN ports from kafka cluster.

What am i supposed to do ? and please let me know what i am missing now about docker (because i am really new to docker :( )

This is yaml configuration

---
version: '2'
services:
  zookeeper-1:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_CLIENT_PORT: 22181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
      ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
    network_mode: host
    extra_hosts:
      - "moby:127.0.0.1"

  zookeeper-2:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_SERVER_ID: 2
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
      ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
    network_mode: host
    extra_hosts:
      - "moby:127.0.0.1"

  zookeeper-3:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_SERVER_ID: 3
      ZOOKEEPER_CLIENT_PORT: 42181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_INIT_LIMIT: 5
      ZOOKEEPER_SYNC_LIMIT: 2
      ZOOKEEPER_SERVERS: localhost:22888:23888;localhost:32888:33888;localhost:42888:43888
    network_mode: host
    extra_hosts:
      - "moby:127.0.0.1"

  kafka-1:
    image: confluentinc/cp-kafka:latest
    network_mode: host
    depends_on:
      - zookeeper-1
      - zookeeper-2
      - zookeeper-3
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: localhost:22181,localhost:32181,localhost:42181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:19092
    extra_hosts:
      - "moby:127.0.0.1"

  kafka-2:
    image: confluentinc/cp-kafka:latest
    network_mode: host
"docker-compose.yml" 83L, 2321C

score 1 · Accepted Answer · answered Jan 06 '20 at 20:05

1

If you are running Kafka (and Zookeeper) on your host, then, no, you don't need Docker at all.

You should not use localhost in any connection string from the container, rather use the external service names of the containers and remove network_mode from the configurations.

Note: multiple brokers/zookeepers on one machine isn't helping anything.

or search data stored in the topic

You shouldn't use zookeeper to do that

answered Jan 06 '20 at 20:05

OneCricketeer

179,855
19
132
245

Using the external service names of the containers and removing network_mode exactly working for me. Thanks a lot :) I understand the note which is explaining multiple brokers/zookeepers on one machine isn't helping anything, after testing docker kafka clustering in my local then i will divide those containers to each AWS machine. Thanks :) – Jaeho Lee Jan 07 '20 at 00:59
If running Docker on AWS, you might as well just use Kubernetes Helm Charts. If you don't want to manage that, just use MSK, Confluent Cloud, or other hosted Kafka – OneCricketeer Jan 07 '20 at 01:21
what if i just use AWS as an each one docker container host without Kubernetes. Do you think Kafka is still possibly working with ? – Jaeho Lee Jan 08 '20 at 07:55
That's exactly what the hosted offerings provide. Kafka works better outside a container, in my opinion – OneCricketeer Jan 08 '20 at 13:32

why should i use docker image "confluentinc/kafka" to kafka cluster?

1 Answers1