3

I am running a Kafka producer in a local machine using my Intellij IDE & the producer will be producing a million records. While doing so, I want to capture the producer metrics in the below way:

enter image description here

I am aware about JMX port for kafka & I did try setting the Kafka JMX port to 9999. But I am not sure if we can get the metrics using JConsole or JVisualVM in the above way that I am expecting.

Can any one suggest any idea as to how can this be achieved?

Jestino Sam
  • 526
  • 2
  • 12
  • 31
  • look at this question. https://stackoverflow.com/questions/53985430/gathering-kafka-producer-metrics-using-jmx – Kenry Sanchez May 07 '19 at 05:35
  • @KenrySanchez I had a look at the post. I had added a comment there as well. It's just that I am unable to figure how to enable the producer metrics or rather how can I get it in the above manner – Jestino Sam May 07 '19 at 05:39
  • I think Elastic Stack can do that. Kafka + Elastic Stack. – Jin Lee May 07 '19 at 06:28

1 Answers1

7

In addition of JMX, the official Kafka clients also expose their metrics via the Java API, see the metrics() method to retrieve them all.

For example, to print all the metrics' names and values:

for (Entry<MetricName, ? extends Metric> entry : producer.metrics().entrySet()) {
    System.out.println(entry.getKey().name() + " : " + entry.getValue().metricValue());
}

Out of all the metrics, you are probably interested in outgoing-byte-rate, request-total and request-rate.

Mickael Maison
  • 25,067
  • 7
  • 71
  • 68
  • Thanks for this API ! I wasn't aware about it. However I am trying to accumulate stats checking for 10k records, then 100k and so on as mentioned in the question. I tried printing the metrics, but its giving me an overall metric while I wanted it at specific intervals. How can this be achieved with the API? – Jestino Sam May 07 '19 at 18:46
  • 1
    You'll have to call it at specific intervals explicitly, it's easy to count records. – Mickael Maison May 08 '19 at 08:08
  • It would be great if you can help me with posting a snippet of calling it at specific intervals. – Jestino Sam May 08 '19 at 09:39
  • Is there any way to get the ````Request total time```` from ````kafka.network:type=RequestMetrics,name=TotalTimeMs,request={Produce|FetchConsumer|FetchFollower}```` using the Metrics java API? Thanks – Jestino Sam May 11 '19 at 19:50