Is there any Kafka Monitoring API in Java?

Question

I am trying to write a Java code to get consumer offsets and lag for our kafka cluster. Is there a Java API to get this information ?

score 8 · Accepted Answer · answered Jun 22 '16 at 05:23

You can get this info if you use Kafka Client libraries and their APIs. That said, there are tools that can give you this info and a lot more if you are after Kafka metrics/monitoring. Burrow from Linkedin and SPM for Kafka from Sematext are two good ones to look at.

score 2 · Answer 2 · answered Sep 14 '15 at 12:37

2

You can use Java Management Extensions (JMX) to monitor your KAFKA.
Kafka already has inbuilt JMX, you just need to enable it -

In the kafka setup, edit “kafka-run-class.sh” script file by adding the following line to it:

KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false "

Also, edit the “kafka-server-start.sh” script file present in the kafka setup to set the JMX port to 9999 by adding the following line:

export JMX_PORT=${JMX_PORT:-9999}

answered Sep 14 '15 at 12:37

Bector2

152
8

tnx, but i like to have a beautiful API – user3359139 Sep 15 '15 at 09:28
Use Ganglia on top of it. You will get all the metrics which you wish to see. – Bector2 Sep 16 '15 at 08:50

score 0 · Answer 3 · answered Sep 09 '15 at 01:03

0

You should find what you need in this project: https://github.com/apache/storm/tree/master/external/storm-kafka/src/jvm/storm/kafka

There you can see the kafka.javaapi.consumer.SimpleConsumer and how it's being used. Use Maven to pull all the dependencies from the Maven repo and you should see a kafka_2.10-0.8.2.1.jar file that provides the Kafka API calls that you need.

The github project referenced above should give you extra pointers on how the API is used.

answered Sep 09 '15 at 01:03

Derar

63
1
3

Unfortunately, SimpleConsumer let me to know offset fro one client not all consumers on on topic. I need all offsets. in addition it present nothing regarding lag. However, i need this API at Producer side not Consumer side. – user3359139 Sep 09 '15 at 04:47
@user3359139 the same API has the package package kafka.producer which has all you need for Producer and Consumer classes. – Derar Sep 09 '15 at 19:37
OK, then let me know the API that gives lag size for a topic and consumer group... – user3359139 Sep 10 '15 at 10:14

score 0 · Answer 4 · answered Nov 06 '15 at 07:57

0

Yes ,there is an application which write in java to track the offsets for all the consumer groups in a Kafka cluster.You can get the result in Json or html, you can also configure the app to push the offset metrics at a regular interval to a StatsD server.

here is the project:

https://github.com/Symantec/kafka-monitoring-tool

the monitor result :

answered Nov 06 '15 at 07:57

heaven

25
6

This link is dead and not updated even on Symantec's page! Is it not in use? – Black Diamond Jul 31 '19 at 10:35

score -1 · Answer 5 · edited May 26 '16 at 08:02

The offsets of group are stored in zookeeper, you need a zookeeper instance,a nd then use the method zk.getChildren(path,false) to get node info and use method zk.getData(path,false,null) to get the offset info.

Set path="/consumers" you can get the consumers group list.

Set path="/consumers/xxx group/offset/xxx topic/xxx partition" you can get the offset.

Is there any Kafka Monitoring API in Java?

5 Answers5