0

Problem

I am thinking about implementing binary search to find a starting offset for time-based event replaying. In order to do so I was thinking about using EARLIEST to find the beginning offset and LATEST to find the latest offset. After that I could implement binary search to find what offset I need to start replaying from.

Question

I was wondering how efficient seeking to EARLIEST and LATEST is and how it is implemented. Do they simply just use the znode timestamp in your topic's directory and find the file with the latest timestamp to look at? That would be my guess but I'm just shooting into the dark there.

Thank you in advance!

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Niraj
  • 113
  • 3
  • 10

1 Answers1

3

If you use Kafka version 0.10 problem has already been solved for you. Since Kafka 0.10 each message can contain timestamp that can be used for accurate searching. Kafka maintains an index based on timestamps which allows users to seek to offset based on time.

Kafka 0.10
You can seek to offset by given timestamp using this method KafkaConsumer#offsetsForTimes

Kafka 0.9 and earlier
There is no timestamp in messages. You can't seek accurately, but you can at least get an approximate offset before given timestamp. Then you need to use Kafka Simple API. I recommend to read more about this topic in this blog post about A Closer Look at Kafka OffsetRequest

vanekjar
  • 2,386
  • 14
  • 23
  • That helps a lot! Thank you. For now I implemented binary search as we aren't at 0.10.0.1 yet unfortunately. When we get there I plan on using offsetsForTimes! – Niraj Nov 13 '16 at 23:40