0

My docker-compose file

version: '3'
services:
  zookeeper:
    container_name: zookeeper
    image: confluentinc/cp-zookeeper
    ports:
      - "32181:32181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SYNC_LIMIT: 2
    
  kafka:
    container_name: kafka
    image: confluentinc/cp-kafka
    ports:
      - "9094:9094"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: localhost:32181
      KAFKA_LISTENERS: INTERNAL://localhost:9092,OUTSIDE://localhost:9094
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://localhost:9092,OUTSIDE://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      ES_JAVA_OPTS: "-Xms512m -Xmx3000m"

enter image description here

and Producer code is

var kafka = require('kafka-node'),
    Producer = kafka.Producer,
    KeyedMessage = kafka.KeyedMessage,
    client = new kafka.KafkaClient({kafkaHost:"localhost:9094"}),

    producer = new Producer(client),
    km = new KeyedMessage('key', 'message'),
    payloads = [
        { topic: 'topic1', messages: 'hi', partition: 0 },
        { topic: 'topic1', messages: ['hello', 'world', km] }
    ];

client.createTopics(topicsToCreate, (error, result) => {
                
                echo ("------------------------KAFAK--------------------")
                console.log(error);
                console.log(result);            
            });

Getting below error, while creating the topic before sending pay-load to topics

{ BrokerNotAvailableError: Broker not available (loadMetadataForTopics)
    at new BrokerNotAvailableError (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\errors\BrokerNotAvailableError.js:11:9)
    at KafkaClient.loadMetadataForTopics (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:891:21)
    at KafkaClient.loadMetadata (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:876:8)
    at KafkaClient.getController (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:267:8)
    at KafkaClient.sendControllerRequest (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:1219:8)
    at KafkaClient.createTopics (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:935:8)
    at C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\index_behind_kong.js:60:11
    at C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\baseClient.js:370:18
    at KafkaClient.loadMetadataForTopics (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\kafkaClient.js:891:12)
    at RetryOperation._fn (C:\wamp64\www\ws-proxy\HOZ-KOG-WebSocket_NodeJS\node_modules\kafka-node\lib\baseClient.js:360:12) message: 'Broker not available (loadMetadataForTopics)' }
undefined

I have gave the delay of 5 seconds before calling to createtopic method, however no luck found.

Kindly assist.

Vipin Kumar R. Jaiswar
  • 741
  • 6
  • 6

1 Answers1

0

Few things things i would try out

  1. KAFKA_ZOOKEEPER_CONNECT: localhost:32181 should change to KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181

  2. For 'INTERNAL' Listeners : change from INTERNAL://localhost:9092 to INTERNAL://kafka:9092

  3. You have missing dependson for zookeeper in kafka container

  4. Hope you have made sure that no other service is running on port 9094 and 32181 on your localhost.

  5. Try connecting using kafka-topics "sh or bat" script before going with java application

  6. I don't think it would be an issue but still you can try adding a common network tag under both services and also under the compose file like : networks: - kafka-network

Helpful links : https://www.confluent.io/blog/kafka-listeners-explained/ (check for section HOW TO: "Connecting to Kafka on Docker" )

user3887600
  • 138
  • 1
  • 8