3

I am unable to use JMX exporter to expose kafka metrics. Can you look at my steps and correct me where needed.? I am following steps here to enable kafka with JMX exporter.

Following are step by step instruction I followed

#get kafka
wget kafka_2.11-2.0.0

# Download Prometheus JMX exporter:
sudo wget -P /opt/kafka/prometheus/ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.0/jmx_prometheus_javaagent-0.3.0.jar
sudo wget -P /opt/kafka/prometheus/ https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml

#Edit Prometheus JMX exporter config file; and append following lines 
echo  “- pattern : kafka.producer<type=producer-metrics, client-id=(.+)><>(.+):\w* name: kafka_producer_$2” >> /opt/kafka/prometheus/kafka-0-8-2.yml
echo  “— pattern : kafka.consumer<type=consumer-metrics, client-id=(.+)><>(.+):\w* name: kafka_consumer_$2” >> /opt/kafka/prometheus/kafka-0-8-2.yml
echo  “— pattern : kafka.consumer<type=consumer-fetch-manager-metrics, client-id=(.+)><>(.+):\w* name: kafka_consumer_$2” >> /opt/kafka/prometheus/kafka-0-8-2.yml


#start zookeeper in terminal 0
/opt/kafka/bin/zookeeper-server-start.sh config/zookeeper.properties


#start kafka broker in terminal 1
KAFKA_HEAP_OPTS=’”-Xmx1000M -Xms1000M”’ 
KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml” 
JMX_PORT=7071
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties  


#start kafka consumer in terminal 2
KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7072:/opt/kafka/prometheus/kafka-0–8–2.yml” 
JMX_PORT=7072
/opt/kafka/bin/kafka-console-consumer.sh — bootstrap-server 0.0.0.0:9092 — topic test — from-beginning

#start kafka producer in terminal 3
KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7073:/opt/kafka/prometheus/kafka-0–8–2.yml” 
JMX_PORT=7073
/opt/kafka/bin/kafka-console-producer.sh — broker-list 0.0.0.0:9092 — topic test

After above steps zookeeper and kafka is running fine. I can type in producer terminal a message and it is received on consumer console. However no kafka metrics is visible on Prometheus. To debug this I checked ports 7071/2/3 by 

netstat -tlnp | grep 7071
netstat -tlnp | grep 7072
netstat -tlnp | grep 7073

which resulted in blank response; this means no service is using above ports. I feel like JMX exporter is not enabled correctly.

Can you help me with above issues?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
ankit patel
  • 1,399
  • 5
  • 17
  • 29

2 Answers2

5

From the looks of your question, you put the variables on their own lines, while the blog has them on the same line...

e.g. This is how to start the Kafka server

KAFKA_HEAP_OPTS='"-Xmx1000M -Xms1000M"' KAFKA_OPTS='-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml' JMX_PORT=7081 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

Otherwise, you need to export the variables so the sub-process will pick them up like you did in your previous question, which seemed to be working fine for exposing the metrics 

export KAFKA_HEAP_OPTS='"-Xmx1000M -Xms1000M"'
export KAFKA_OPTS='-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml'
export JMX_PORT=7081
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

Note: The blog you linked to doesn't use JMX_PORT, but those ports cannot be the same as the exporter.

I would also suggest at least downloading a version newer than 0.3 - https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/

and using the configs for Kafka 2.0 - https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml

Sidenote: netstat -tlnp | grep 707 would show you all them at once

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • i used 0.12.0 java agent with kafka 2.0.0 yaml but now i get "Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7071; nested exception is: java.net.BindException: Address already in use (Bind failed) jdk.internal.agent.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 7071; nested exception is: java.net.BindException: Address already in use (Bind failed) " error even-though "lsof -i:7071" and "netstat -tlnpu | grep 70" shows nothing – ankit patel Oct 29 '19 at 18:08
  • Try using a different port? If lsof doesn't show anything, then I'm not sure what the issue would be – OneCricketeer Oct 29 '19 at 18:11
  • 1
    Oh. Set `JMX_PORT` to not match the agent, or just remove it... Prometheus Agent starts a web server. JMX is not over HTTP – OneCricketeer Oct 29 '19 at 18:17
  • Bingo it worked i can see a process listening on 6071 now. – ankit patel Oct 29 '19 at 18:23
  • netstat -tlnpu |grep 60 tcp6 0 0 :::6071 :::* LISTEN 35522/java – ankit patel Oct 29 '19 at 18:23
  • now i want to see all Broker metrics on Prometheus; i have got your comment here https://stackoverflow.com/questions/58033823/configuring-kafka-with-jmx-exporter-centos-7/58602767#58602767 i will work on gettin prometheous working on this node; Via docker container i am not able to see any broker metrics – ankit patel Oct 29 '19 at 18:25
  • Kafka docker container? – OneCricketeer Oct 29 '19 at 18:40
  • 1
    I am closing this question since i get all broker metrics on curl -s :7071 | grep -i kafka. thanks cricket I will open a new one for porting these metrics to Prometheus – ankit patel Oct 30 '19 at 18:51
  • can you look at this question https://stackoverflow.com/questions/58634019/jmx-exporter-and-prometheus – ankit patel Oct 30 '19 at 22:24
2

thank you cricket-007 for your help. I am listing steps i followed here for simplicity

wget -q -O /tmp/kafka.tgz https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz
tar -xf /tmp/kafka.tgz --directory /opt/kafka --strip 1
rm -f /tmp/kafka.tgz

wget -q -O /tmp/kafka.tgz https://archive.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz
mkdir /opt/kafka
tar -xf /tmp/kafka.tgz --directory /opt/kafka --strip 1
rm -f /tmp/kafka.tgz
sudo wget -P /opt/kafka/prometheus/ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar
wget https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-2_0_0.yml
cd kafka
export KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.12.0.jar=7071:/opt/kafka/prometheus/kafka-2_0_0.yml"
export KAFKA_HEAP_OPTS="-Xmx1000M -Xms1000M"
mv ../kafka-2_0_0.yml prometheus/
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
netstat -tlnpu |grep 70

tcp6       0      0 :::7071                 :::*                    LISTEN      209455/java         
udp6       0      0 :::40705                :::*      

curl -s localhost_or_IP:7071 | grep -i kafka
long list of metrics will be dumped on stdout                          -
ankit patel
  • 1,399
  • 5
  • 17
  • 29