1

Here is what I am trying

Collection <TopicPartition> partitions = consumer.partitionsFor(topic).stream();

And also how to indicate you've hit the end or there isn't anymore messages to consume. If the offset doesn't match the broker's end offset at the time how to do that.

Any suggestions.

Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
Reddy
  • 11
  • 1
  • 4

1 Answers1

1

In order to get the latest offset you can either use the command line: 

./bin/kafka-run-class.sh kafka.tools.GetOffsetShell \
    --broker-list localhost:9092 \
    --topic topicName

or programmatically in Java: 

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProperties)) {
    consumer.subscribe(Arrays.asList("topicName"));
        Set<TopicPartition> assignment;
        while ((assignment = consumer.assignment()).isEmpty()) {
            consumer.poll(Duration.ofMillis(500));
        }
        consumer.endOffsets(assignment).forEach((partition, offset) -> System.out.println(partition + ": " + offset));
}

Now if you want to force the consumer to start consuming from the latest offset, you can either use the following property: 

props.put("auto.offset.reset", "latest"); 
// or props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");

or force it to consume from latest offset using

consumer.seekToEnd();
Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
  • Could you also please suggest on how do we know if the consumer queue is full?and do we have to perform some retry logic to keep fetching messages for a certain amount of time in case of missed messages? could you post if that can be done programmatically in java.Thanks! – Reddy Apr 22 '20 at 19:31
  • @Reddy I am not sure if you can actually check if there are no messages left. Kafka is all about real-time data streaming so this question doesn’t make a lot of sense to me. – Giorgos Myrianthous Apr 22 '20 at 19:32
  • @ Giorgos Myrianthous Okay.Here is what I am doing to get a record Optional> record = consumer.nextRecord(timeout); if (record.isPresent()) then returning record.get() else null.But the problem is looks like something is wrong when getting the messages, seeing records are being missed.Is it something due to consumer's internal queue?Not sure though about how consumer's internal queue works? – Reddy Apr 22 '20 at 19:52
  • @Reddy It would be much easier to ask a new question and share your full code with the new problem that you have. It’s hard to tell by just seeing a single line of your code. – Giorgos Myrianthous Apr 22 '20 at 19:54
  • 1
    Sure I will add new question. – Reddy Apr 22 '20 at 19:55
  • @Reddy Share the link and I’ll take a look later. – Giorgos Myrianthous Apr 22 '20 at 19:56