3

I am building an Apache Kafka consumer to subscribe to another already running Kafka. Now, my problem is that when my producer pushes message to a server...my consumer does not receive them. Here I give Producer code,

        Properties properties = new Properties();
        properties.put("metadata.broker.list","Running kafka ip addr:9092");
        properties.put("serializer.class","kafka.serializer.StringEncoder");
        ProducerConfig producerConfig = new ProducerConfig(properties);
        kafka.javaapi.producer.Producer<String,String> producer = new kafka.javaapi.producer.Producer<String, String>(producerConfig);
        String filePath="filepath";
        File rootFile= new File(filePath);
        Collection<File> allFiles = FileUtils.listFiles(rootFile, CanReadFileFilter.CAN_READ, TrueFileFilter.INSTANCE);
        for(File file : allFiles) {
            StringBuilder sb = new StringBuilder();
            sb.append(file);
            KeyedMessage<String, String> message =new KeyedMessage<String, String>(TOPIC,sb.toString());
            System.out.println("sending msg from producer.."+sb.toString());
            producer.send(message);
        }
           producer.close();

Here Consumer code,

         properties.put("bootstrap.servers","Running zookeaper ip addr:2181");
         properties.put("group.id","test-group");
         properties.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
         properties.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
         properties.put("enable.auto.commit", "false");

         KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(properties);
         consumer.subscribe(Collections.singletonList(topicName)); 
         while (true) {
                ConsumerRecords<String, String> records = consumer.poll(100);
                for (ConsumerRecord<String, String> record : records)
                {
                    System.out.println("topic = "+record.topic());
                    System.out.println("topic = "+record.partition());
                    System.out.println("topic = "+record.offset());
                }
                try {
                  consumer.commitSync(); 
                } catch (CommitFailedException e) {
                    System.out.printf("commit failed", e) ;
                }
            }

I use this dependency:

    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.11</artifactId>
        <version>0.10.0.0</version>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>0.10.1.0</version>
    </dependency>

I get all information from that link:
https://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

When we running consumer, we didn't get any notification from the consumer side. Please give me any idea.

saloua
  • 2,433
  • 4
  • 27
  • 37
Jitendra kumar
  • 49
  • 1
  • 8
  • try to figure out where is the problem: on consumer or producer size. For this: check offsets in topic. It can be done from command line – Natalia Mar 22 '17 at 13:40
  • You are running it as a jar file on cluster?..please verify your zookeeper port. – ketankk Mar 22 '17 at 17:29
  • @Natalia: I am able to post the messages through producer. I can see the message number increasing along with the log size..but offset is not increasing... – Jitendra kumar Mar 23 '17 at 06:10
  • @KetanKeshri - I am running in eclipse as of now.. – Jitendra kumar Mar 23 '17 at 06:11
  • You check that your kafka is working correctly by using the command line tools.For that you need to produce messages with the command line producer `kafka-console-producer` and then make sure that the messages get consumed with the command line consumer `kafka-console-consumer` https://kafka.apache.org/quickstart#quickstart_send – saloua Mar 23 '17 at 11:01
  • @saloua- Yes both Kafka producer and consumer running perfectly but still not getting any message in consumer side – Jitendra kumar Mar 23 '17 at 12:53
  • @Jitendra kumar- Please try once by running it locally on the cluster where kafka+zookeeper is installed by making a jar of your code. – ketankk Mar 23 '17 at 19:07
  • 1
    @Jitendrakumar Do you mean that you are not getting any message from the command line consumer `kafka-console-consumer` ? – saloua Apr 07 '17 at 19:14
  • If the command line consumer is not getting any produced message then you certainly have an error in the configuration of your brokers. – saloua Apr 07 '17 at 19:24

2 Answers2

0

For producer:

   properties.put("metadata.broker.list","Running kafka ip addr:9092");

I guess, this should be "bootstrap.servers".

For consumer:

properties.put("bootstrap.servers","Running zookeaper ip addr:2181");

bootstrap.servers must point to a broker, not to ZK.

The "problem" is, that the consumer will just wait for a broker but not fail if there is no broker at the specified host/port.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
  • We are running both broker and zookeeper in the same ip. Its a single node installation. Hence gave the same ip.Do I have to run zookeeper and broker in different vms?? – Jitendra kumar Mar 23 '17 at 06:09
  • You don't need to run on different servers -- but it's recommended. Anyway, ZK and broker use different ports, and `2181` is ZK default port -- so I guess you need up point to broker port (default: `9092`) – Matthias J. Sax Mar 23 '17 at 18:44
  • @ Matthias J. Sax - did in same but still not getting any message in consumer side – Jitendra kumar Mar 24 '17 at 05:12
  • Just updated my answer. Is your producer setting correct? You should check the broker /client logs if producer/consumer do connect successfully to the broker, too. – Matthias J. Sax Mar 24 '17 at 17:23
0

I'm a newb at Kafka and Java, but i'll like to suggest the following approach

  • Verify that the producer is actually writing to the topic using the following command /usr/bin/kafka-avro-console-consumer --new-consumer --bootstrap-server localhost:9092 --topic KumarTopic --from-beginning.
  • If it is, you'll probably need to focus on your consumer code. Confluent's guides are pretty helpful.
Zigmaphi
  • 15
  • 5
  • @Zigmaphi-Thnaks for comment, I already checked. Producer writing in topic perfectly and consumer also running but still not getting any message – Jitendra kumar Mar 27 '17 at 14:21