Getting latest message timestamp in my Kafka queue

Question

I have Kafka 0.10.0, which, if I understand correctly, adds timestamps to all of the messages. For monitoring purposes, I want to pull out the timestamp of the newest message for a given topic. I did not see an API field for it in any of the Python libraries I have looked at.

In a regular consumer from the latest offset, you just get the timestamp. However, how do you define "latest" when you start getting more than one event per second? As soon as you query the data, it's stale... — OneCricketeer, Dec 19 '18 at 15:19
@cricket_007 This is for monitoring purposes. I don't care what exactly the contents of the latest message is, just what the timestamp is on it (to some rough granularity). — Konstantin Tarashchanskiy, Dec 19 '18 at 17:25
Four years later... you can use `consumer.end_offsets()` now — kbridge4096, May 26 '22 at 07:01

score 3 · Accepted Answer · edited Jan 20 '21 at 09:51

There is not straightforward method to get the latest message Timestamp from Kafka topics. But the work around is using kafka consumer, and use seek_to_end() to seek the most available offset for partitions.

consumer.seek_to_end()
for message in consumer:
    print(message.timestamp)

You can refer the details here :
https://kafka-python.readthedocs.io/en/master/apidoc/KafkaConsumer.html#kafka.KafkaConsumer.seek_to_end

Getting latest message timestamp in my Kafka queue

1 Answers1