0

I have set up my kafka producer a fresh following the tutorial mentioned here: https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-14-04

I am pushing some events to the producer using cron and following script on server with IP: 1.2.3.4 

#!/usr/bin/env python
import threading, logging, time
import multiprocessing
import requests
import datetime
import json
from kafka import KafkaProducer

class CheckApis():
    apisList = {"a": "https://test.eng.com/"}
    kafkaProducer = "1.2.3.4:9092"
    kafkaTopic = "sometopic"
    producer = KafkaProducer(bootstrap_servers=kafkaProducer)
    for key, value in apisList.items():
        headers = {};
        response = requests.request("GET", value, headers=headers)
        message = {"app_name": key, "status": response.status_code, "message": "none", "timestamp": str(datetime.datetime.utcnow())}
        producer.send(kafkaTopic, json.dumps(message).encode('utf-8'));
        print (response.text)
        print (response.status_code)
    producer.close()

This works well, and I can see the pushed events using this command:

~/kafka/bin/kafka-console-consumer.sh --zookeeper 1.2.3.4:2181 --topic sometopic --from-beginning

But when I try to use these events remotely from other server (my laptop), it fails with error:

error: Some error occured while listening to kafka events Error: connect ECONNREFUSED 5.6.7.8:9092 (Some different IP here not 1.2.3.4)

Here is my consumer code (in node js using kafka-node):

var ConsumerGroup = require('kafka-node').ConsumerGroup;
    var healthConsumerOption = {
        host: '1.2.3.4:2181',
        autoCommit: true,
        groupId: os.hostname(),
        sessionTimeout: 15000,
        protocol: ['roundrobin'],
        fromOffset: 'latest'
    };
    var healthConsumerGroup = new ConsumerGroup(healthConsumerOption, healthTopics);
        listenHealthEventsKafka: function(connections){
            try{
                healthConsumerGroup.on('error', onError);
                healthConsumerGroup.on('message', onMessage);
                healthConsumerGroup.on('connect', function(){
                    logger.info("Health consumer group is ready. ")
                });

                function onMessage(message){
                    var jsonData = JSON.parse(message.value);
                    console.log(message);
                };

                function onError(error){
                    logger.error("Some error occured while listening to kafka events " +error);
                }

                process.once('SIGNINT', function(){
                    async.each([healthConsumerGroup], function(consumer, callback){
                        logger.info("Closing the kafka health consumer process ");
                        consumer.close(true, callback);
                    });
                })
            }catch(error){
                logger.error("Could not connect to kafka events for build " +error);
            }
        }

Do I need to do additional configuration on Kafka server (server.properties) to allow remote access.. or I am doing anything wrong? Please help.

undefined
  • 3,464
  • 11
  • 48
  • 90
  • So what is this second IP that it complains about not being able to connect to? It must have been set somewhere... – Thilo May 04 '18 at 11:08
  • What is the advertised hostname for the brokers? Clients need to be able to route to that. – Thilo May 04 '18 at 11:09
  • I dont know from where this 2nd IP is coming, I have never mentioned it anywhere in my code.. (its not even IP of my laptop) @Thilo – undefined May 04 '18 at 11:12
  • My guess is that is the IP of the Kafka broker. Which is configured at the broker-side. The client gets this from Zookeeper or a bootstrap broker. If this is an internal network IP, you cannot just access it from your network without some deep routing or VPN magic. – Thilo May 04 '18 at 11:14
  • See also https://stackoverflow.com/questions/27499527/access-kafka-broker-in-a-remote-machine-error?rq=1 – Thilo May 04 '18 at 11:16

1 Answers1

6

On your Kafka broker server.properties you need to set advertised.listeners to the external IP so that clients can correctly connect to it. Otherwise they'll try to connect to the internal IP (since advertised.listeners will default to listeners unless explicitly set)

Ref: https://kafka.apache.org/documentation/#brokerconfigs

See also https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/

Robin Moffatt
  • 30,382
  • 3
  • 65
  • 92