17

I am new to KAFKA and I know that this question has been asked multiple times on stack overflow but none of the solutions worked for me so here I am trying my luck with asking the same question again. I have downloaded and installed KFKA on Centos7 VM. The VM is on my laptop. When I run the KAFKA producer and consumer from the command line, it works fine. Next step, I wanted to create a Java Producer but it always timeout with the following exception.

    Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
    at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1186)
    at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:880)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:803)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:690)
    at com.soft.teradata.KafkaProducerExample.runProducer(KafkaProducerExample.java:40)
    at com.soft.teradata.KafkaProducerExample.main(KafkaProducerExample.java:55)
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

The Java code for the producer is:

package com.soft;
import java.util.Properties;
import java.util.concurrent.Future;
import org.apache.kafka.clients.producer.Producer;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
public class SimpleProducer {
    public static void main(String[] args) throws Exception {
        try {
            String topicName = "Hello-Kafka";
            Properties props = new Properties();
            props.put("bootstrap.servers", "192.168.xxx.xxx:9092");
            props.put("acks", "all");
            props.put("retries", 1);
            props.put("batch.size", 16384);
            props.put("linger.ms", 1);
            props.put("buffer.memory", 33554432);
            props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
            props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
            Producer<String, String> producer = new KafkaProducer<String, String>(props);
            Future<RecordMetadata> f=producer.send(new ProducerRecord<String, String>(topicName, "Eclipse"));
            System.out.println("Message sent successfully");
            producer.close();
        } catch (Exception e) {
            System.out.println(e.getMessage());
        } 
            System.out.println("Successful");

    }
}

For bootstrap.server, i have even tried the following:

props.put("bootstrap.servers", "PLAINTEXT://192.168.xxx.xxx:9092");

Please note that I am executing the java code from Eclipse on my laptop and KAFKA is installed on a CENTOS7 VM on my laptop. 192.168.xxx.xxx is the IP address of the CENTOS7 VM. I have noticed that 192.168.xxx.xxx:9092 (telnet 192.168.xxx.xxx 9092) is inaccessible from my laptop. I added the port to firewall but still no success.

firewall-cmd --zone=public --add-port=9092/tcp --permanent
        firewall-cmd --reload
        firewall-cmd --list-ports

The version of KAFKA is 2.12-2.0.0 and I have added the following jars to my Eclipse Classpath:

  • kafka-clients-2.0.0.jar
  • lz4-java-1.4.1.jar
  • slf4j-api-1.7.25.jar
  • snappy-java-1.1.7.1.jar

Thanks a lot for the help in advance :)

Regadrs, DIRSHAH.

Fawad Shah
  • 1,680
  • 4
  • 15
  • 21
  • 1
    Can you run it if Kafka is running locally, on the same servers as the java code you're running, so you don't have to traverse the network? Usually these metadata timeout errors in Kafka are simply misconfigured network interfaces. – mjuarez Oct 14 '18 at 04:30
  • Thanks a lot for your response. Yes, it works when I create an executable jar file and execute the jar file from within the Centos7 VM. Cento7 VM is where Kafka is also running. I had to change "192.168.xxx.xxx:9092" to "localhost:9092" though. Please note that I am unable telnet from my laptop to CENTOS7 VM on "192.168.xxx.xxx:9092". So I dont know how to make this port visible or available to the World. – Fawad Shah Oct 14 '18 at 14:16
  • That's your problem right there. Port 9092 has to be visible to all cluster nodes so they can exchange data, and also to all producers/consumers. – mjuarez Oct 15 '18 at 00:55
  • Thanks again for your response. But how can I do that? I tried to something like below but its not working. firewall-cmd --zone=public --add-port=9092/tcp --permanent – Fawad Shah Oct 15 '18 at 11:40

4 Answers4

10

This error can indicate that the topic doesn't exist, so you may want to double check your topicName = "Hello-Kafka".

Though this isn't exactly a deep answer, it seems to be a common problem, see also https://github.com/dpkp/kafka-python/issues/607

ynux
  • 1,280
  • 14
  • 21
2

I faced the same issue. You need to advertise the hostname/ip of Kafka broker to be reachable from Kafka Producer pc.

 kafka-server-start.sh config/server.properties --override  advertised.listeners=PLAINTEXT://<accessible-hostname>:9092
Ahmad Alhilal
  • 444
  • 4
  • 19
0

I had the same problem running the cluster locally, I checked if the topic was created in the broker:

./kafka-topics.sh --list  --bootstrap-server=localhost:9092

mine was created so I checked the code again and I realized I had a trailing space in the topic name when sending the messages, I removed the space and it worked.

Josepil
  • 13
  • 6
-6

Same issue, you will find some port can work, some not.

Just change it to 19092 will works, coz rabbitMQ using 15672 works well, i guess the reason was some port taken by OSX

user1811060
  • 19
  • 2