14

Is it possible to retrieve messages in Kafka console conumer for a particular timestamp range?

For example kafka messags between 08:00 to 09:00 yesterday.

Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156

2 Answers2

15

You can use kcat for consuming messages between two timestamps:

kcat -b localhost:9092 -C -t mytopic -o s@1568276612443 -o e@1568276617901

where

  • s@ denotes the starting timestamp in ms
  • e@ denotes the ending timtestamp in ms (non-inclusive)
Bruno Saboia
  • 332
  • 3
  • 18
Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156
  • `s@1568276612443` , is this milliseconds from start of the day?? –  Mar 31 '20 at 09:02
  • @Hummingbird It's a timestamp. You can use any timestamp you wish. `s@` denotes the starting timestamp and `e@` denotes the ending timestamp. – Giorgos Myrianthous Mar 31 '20 at 09:29
  • yes,but the number `1568276612443`, what is this?? a millisecond or nanosecond and if yes , from when does it began?? start of the day or start of some particular day?? –  Mar 31 '20 at 11:14
  • @Hummingbird OK. What is the timestamp interval you want to consume from? – Giorgos Myrianthous Mar 31 '20 at 20:45
  • that's ok , i found the timestamps i wanted . anyways the server i wanted to implement the command does not have kafkacat. –  Apr 01 '20 at 08:06
  • FWIW, this does not seem to work (anymore?). See [this](https://stackoverflow.com/questions/73853753/kafkacat-consume-between-timestamp-giving-wrong-results-when-counting-records/73854735?noredirect=1#comment130412289_73854735) question as well as the corresponding [GitHub issue](https://github.com/edenhill/kcat/issues/399). – filpa Sep 27 '22 at 09:30
2

Yes, you can do it since Kafka version 0.10.1.
Use the function offsetsForTimes in KafkaConsumer:

Look up the offsets for the given partitions by timestamp. The returned offset for each partition is the earliest offset whose timestamp is greater than or equal to the given timestamp in the corresponding partition. This is a blocking call. The consumer does not have to be assigned the partitions.

Ofek Hod
  • 3,544
  • 2
  • 15
  • 26
  • hey ofek , i was actually looking to be done in terminal , using the kafka-console-consumer shell script –  Mar 27 '20 at 10:21
  • 2
    You can use `KafkaConsumer` api just to retrieve the relevant offset, then use `--offset myoffset` in the `kafka-console-consumer`, I don't think there is other solution rather than use `--property print.timestamp=true` in `kafka-console-consumer` and filter the printed time for each message using `grep` in order to fit your range. If this is a good direction for you I can write it in more details. – Ofek Hod Mar 27 '20 at 10:36
  • Or, if your brokers can talk PLAINTEXT, you can use one of Kafka tools to get the offset by timestamp: `kafka-run-class.sh kafka.tools.GetOffsetShell ... --time `. Then, pass that offset to kafka-console-consumer. – mazaneicha Mar 27 '20 at 13:03
  • @OfekHod but wouldn't i be still using an api?? –  Mar 31 '20 at 11:17