1

I am trying to debug a performance issue related to kafka streams stateful application(We use processor API only). The application queries a number of statestores (close to 55). So, in order to find out the reason for slow processing, i was looking for metrics. I found that we can enable jmx metrics for rocksdb in debug mode.

I have set "metrics.recording.level" to debug. But, for some reason, the application is not reporting any metrics. I started getting task level metrics ( which means that the level is debug). Do i need to enable anything else ?

Also, the application runs as a kubernetes pod and creates the state (from changelog) on deployment/restart.

Kafka version : 2.4.1 Kafka client version : 2.4.1

Am I hitting this https://issues.apache.org/jira/browse/KAFKA-9355

Also, are there any native tools which can be used to directly query rocksdb on application pod. btw, i can exec into the pod for any debugging.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
SunilS
  • 2,030
  • 5
  • 34
  • 62

1 Answers1

3

I think you are hitting: https://issues.apache.org/jira/browse/KAFKA-9675

It's already fixed but the fix is not released yet.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
  • Thanks Matthias. Any workaround for this ? Also, are there any native tools to query rocksdb on pod ?. Because when i started monitoring IO on the pod, it could see a consistent IO of 60 MB/s being written to disk (most probably rocksdb). So, it could be related to cache misses or some other tuning for rocksdb. – SunilS Apr 27 '20 at 05:08