3

How to enable jmx metrics for Kafka connectors? I am running connect distributed.

I have enabled jmx metrics for Kafka by enabling JMX_PORT in kafka-run-class.sh.

Now though it has started giving me a lot of Kafka related metrics I still cannot see connector specific metrics mentioned in the below link http://kafka.apache.org/documentation.html#connect_monitoring.

following domains are available

  • JMImplementation
  • com.sun.management
  • java.lang
  • java.nio
  • java.util.logging
  • kafka
  • kafka.cluster
  • kafka.controller
  • kafka.coordinator.group
  • kafka.coordinator.transaction
  • kafka.log
  • kafka.network
  • kafka.server
  • kafka.utils
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Piyush Agarwal
  • 33
  • 1
  • 3
  • Could you include some example of what you have tried so far ? – DarkMukke Nov 30 '18 at 15:15
  • What version of Kafka are you running? Connect metrics weren't added until 1.0 and 1.1 – OneCricketeer Nov 30 '18 at 19:40
  • Also, did you actually edit the run class script? If so, that's not how you enable things – OneCricketeer Dec 02 '18 at 18:00
  • @cricket_007 I m using Confluent Open Source: 5.0.0 which includes kafka 2.0 – Piyush Agarwal Dec 03 '18 at 06:30
  • Can you show an output of `jconsole` or `jmxterm` for the metrics that you do see? – OneCricketeer Dec 03 '18 at 06:39
  • @cricket_007 _#following domains are available_ JMImplementation com.sun.management java.lang java.nio java.util.logging kafka kafka.cluster kafka.controller kafka.coordinator.group kafka.coordinator.transaction kafka.log kafka.network kafka.server kafka.utils – Piyush Agarwal Dec 03 '18 at 07:35
  • 1
    Are you sure you connected to the correct Java instance and not the Kafka broker JMX metrics? Because `kafka.server` should not be there for Connect... Is Connect running on the same machine as other Confluent components? – OneCricketeer Dec 03 '18 at 07:42
  • @cricket_007 Yes, connect is running on same machine. I need to know, what properties do i need to change to enable Connect metrics. and in which file? – Piyush Agarwal Dec 04 '18 at 08:12
  • @PiyushAgarwal could you share how did you manage this? – Gaurang Shah Mar 07 '19 at 03:44
  • @cricket_007 I am using KafkaConnect 3.3.x with Kafka 0.11.0.x. that means I cant get connector specific metrics ? – Ashika Umanga Umagiliya May 31 '19 at 01:23
  • 1
    @Ashika You can upgrade Connect workers independently of the brokers assuming they're on separate servers (which they should be anyway) – OneCricketeer Jun 01 '19 at 04:44
  • @cricket_007 thanks. but according to the Compatibility matrix, I should stick with Confluent 3.3.x If I use Kafka 0.11.x (we don't have control over the Kafka Cluster) https://docs.confluent.io/current/installation/versions-interoperability.html ? – Ashika Umanga Umagiliya Jun 02 '19 at 11:23
  • 1
    @Ashika Maybe you missed this part? *Kafka Connect Workers that are included in Confluent Platform 3.2 and later are compatible with any Kafka broker that is included in Confluent Platform 3.0 and later.* – OneCricketeer Jun 02 '19 at 15:29
  • @cricket_007 Our Kafka Cluster is 0.11 which we don't have any control.Kafka Cluster however, we have full control and we installed on k8s.(5 k8s pods) We used Kafka Connect libraries built from this repo (https://github.com/confluentinc/kafka) branch 3.3.1. We use JDBC Source Connector(3.3.2) and HDFS Sink connector(3.3.2). Do you mean that we can run 3.3.1 Kafka Connect core libraries and use the latest versions of JDBC and HDFS Connector libraries ? – Ashika Umanga Umagiliya Jun 03 '19 at 00:04
  • 1
    @Ashika Yes. You can use Confluent's existing Kafka Connect docker images of any version greater than 3.0 – OneCricketeer Jun 03 '19 at 00:35
  • @cricket_007 As I mentioned, 1) our Kafka Cluster runs vanilla Kafka 0.11 from Apache(not Confluent).This version is fixed and cant be changed. 2) Kafka Connect libraries are from CP-3.3.1 which is based on Kafka 0.11 (connect-api-0.11.0.3-cp3.jar, connect-runtime-0.11.0.3.jar ..etc) 3) I thought the new JMX sending logic is implemented in Kafka Connect core libs (connect-runtime.jar) 1.0.0 and later. 4) So If I use Kafka Connect CP 3.3.1 (Kafka connect core libs 0.11) with latest JDBC and HDFS Sinks(lets say from CP 5.2.0) how does these connectors send these new JMX metrics? – Ashika Umanga Umagiliya Jun 03 '19 at 01:11
  • 1
    @Ashika Confluent doesn't change anything on top of Apache. I don't understand the problem. If you use CP 5.2 Connect docker images with Apache Kafka 0.11 brokers, it should work fine because newer clients can work with older brokers – OneCricketeer Jun 03 '19 at 13:24
  • @cricket_007 tried to install CP 5.2.x images and was giving errors.(related to Kerberos security features).I asked a new question related to this here : https://stackoverflow.com/questions/56519301/kafka-connect-and-kafka-broker-version-compatibility – Ashika Umanga Umagiliya Jun 10 '19 at 00:22

1 Answers1

4

Ask mentioned in the comments - Are you sure you connected to the correct Java instance and not the Kafka broker JMX metrics? Because kafka.server nor kafka.controller should not be there for ConnectDistributed process. (You should look at the main class of the process your connected to)

There's only one property, JMX_PORT, and it is shared by both Kafka brokers and Kafka Connectors (and Zookeeper, Schema Registry, KSQL, etc, etc)... If you do not use different ports for each, then every next process you try to run, will will fail to start, saying a port is already open

If you have the available hardware, or can otherwise isolate components on a single machine (Docker or VM), then that's an easy way to remap ports.

Otherwise, you'd need to do something like 

export JMX_PORT=35000 # pick a number 
zookeeper-server-start...
export JMX_PORT=35001 # pick a number 
kafka-server-start... 
export JMX_PORT=35002 # pick a number 
connect-distributed...

Then, attach to the appropriate ports for each JVM

Then, when you're satisfied with that, I will suggest finding the Confluent Whitepaper on a Production deployment setup, where Connect, at the very least (as with all other producer and consumers), are running off of the actual brokers.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245