How to poll kafka for the messages that has timestamp less than 30 seconds?

Question

I want to poll only the records that have produced less than 30 seconds. Is there any way in the Kafka polling that I can consume only the messages that are produced only less than 30 seconds?

Any suggestions or advice will be really helpful. Thanks in advance.

what will happen to those messages that were not processed due to fully occupied consumers? — Anonymous Duck, Nov 06 '19 at 11:22
Basically I want to drop those messages ( > 30 seconds) during the poll. — Manoj, Nov 06 '19 at 11:29
another way is to set lifespan of the messages, if it's there for more than 30 secs, dispose — Anonymous Duck, Nov 06 '19 at 11:32

score 2 · Answer 1 · answered Nov 06 '19 at 11:20

You could use the offsetsForTimes method which returns you offset whose timestamp is greater or equal to the give timestamp. More information on the official doc here :

https://kafka.apache.org/0101/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#offsetsForTimes(java.util.Map)

After getting the offset you can seek using it and starting to read from there.

https://www.programcreek.com/java-api-examples/index.php?api=org.apache.kafka.clients.consumer.OffsetAndTimestamp

How to poll kafka for the messages that has timestamp less than 30 seconds?

1 Answers1