0

I am basically trying to query offset info using bin\kafka-run-class kafka.tools.GetOffsetShell i am providing broker list and topic info as argument. Ref - GetOffsetShell Utility

But I am facing - Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Here are two SOF posts that touch base this same topic - Retrieve Timestamp based data from Kafka and kafka-run-class throwing java.lang.OutOfMemoryError error. So this question is not about kafka-run-class kafka.tools.GetOffsetShell. So i want to check what other tools are available to query kafka offset info - information like offset and the time when the offset was created.

kafka-consumer-groups.sh does not help my usecase as it does not provide/allow to get timestamp of when the record/offset was created.

Note: I suppose we are using confluent kafka - 5.3.x with SSL/JAAS auth mechnasim

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
joven
  • 371
  • 1
  • 6
  • 17
  • and we are using Avro records – joven Jun 09 '21 at 12:31
  • 1
    You are probably getting an error, first set the following to get the full error, export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G", secondly for sasl you'll need to set KAFKA_OPTS with jaas file and config file with truststore jks for ssl, post the full error for further ass8 – Ran Lupovich Jun 09 '21 at 12:44
  • You can use the kafka console consumer with property print.timestamp=true requesting specific offset adding max messages 1 – Ran Lupovich Jun 09 '21 at 12:46
  • @RanLupovich, i tried `print.timestamp=true` but not getting any timestamp. So do i assume that the broker is not defaulting to set one. – joven Jun 09 '21 at 12:54
  • @RanLupovich basically, i want to consume records between a certain time window. if required i can post a new Q regarding this. – joven Jun 09 '21 at 12:55
  • @RanLupovich, on another look - i see this - `"TimeStamp":"2021-06-09T06:18:13.251-0400"` is this the timestamp you are referring to. I am thinking this is set by the producer application as part of record data – joven Jun 09 '21 at 12:58
  • 1
    The timestamp will be in unix epoch and need to be converted to human readable , it will be in the beginning of the printing of a record like a long number. When I need to filter for your type request usually I am using a free tool named jq which giving a lot of options working with jsons like filtering and selecting ... – Ran Lupovich Jun 09 '21 at 13:04
  • "what other tools" -- Well, you could write your own JVM client that basically imports all the classes that `GetOffsetShell` does if you're not satisfied with the out-of-the-box arguments. Regarding your other comment - [consume between timestamps](https://stackoverflow.com/questions/60882882/how-to-consume-messages-between-two-timestamps-using-kafka-console-consumer) – OneCricketeer Jun 09 '21 at 22:39

0 Answers0