0

I want to enable kafka monitoring and I am starting with single node deployment as test. I am following steps from https://alex.dzyoba.com/blog/jmx-exporter/

i tried following steps; the last command which checks for jmx-exporter HTTP server reports blank. i believe this is the reason, why I am not seeing metrics from kafka.(more on this below)

wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.6/jmx_prometheus_javaagent-0.6.jar
wget https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml   
export KAFKA_OPTS='-javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent-0.6.jar=7071:/etc/jmx-exporter/kafka-0-8-2.yml' 
/opt/kafka_2.11-0.10.1.0/bin/kafka-server-start.sh /opt/kafka_2.11-0.10.1.0/conf/server.properties
netstat -plntu | grep 7071

kafka broker log on console does not have any ERROR message.

i have Prometheus running in a container and http://IP:9090/metrics shows bunch of metrics. when i searched for "kafka" it returned following 

# TYPE net_conntrack_dialer_conn_attempted_total counter
net_conntrack_dialer_conn_attempted_total{dialer_name="kafka"} 79
# TYPE net_conntrack_dialer_conn_closed_total counter
net_conntrack_dialer_conn_closed_total{dialer_name="kafka"} 0
net_conntrack_dialer_conn_established_total{dialer_name="kafka"} 0
# TYPE net_conntrack_dialer_conn_failed_total counter
net_conntrack_dialer_conn_failed_total{dialer_name="kafka",reason="refused"} 79
net_conntrack_dialer_conn_failed_total{dialer_name="kafka",reason="resolution"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="kafka",reason="timeout"} 0
net_conntrack_dialer_conn_failed_total{dialer_name="kafka",reason="unknown"} 79

# TYPE prometheus_sd_discovered_targets gauge
prometheus_sd_discovered_targets{config="kafka",name="scrape"} 1
# HELP prometheus_target_sync_length_seconds Actual interval to sync the scrape pool.
# TYPE prometheus_target_sync_length_seconds summary
prometheus_target_sync_length_seconds{scrape_job="kafka",quantile="0.01"} NaN
prometheus_target_sync_length_seconds{scrape_job="kafka",quantile="0.05"} NaN
prometheus_target_sync_length_seconds{scrape_job="kafka",quantile="0.5"} NaN
prometheus_target_sync_length_seconds{scrape_job="kafka",quantile="0.9"} NaN
prometheus_target_sync_length_seconds{scrape_job="kafka",quantile="0.99"} NaN
prometheus_target_sync_length_seconds_sum{scrape_job="kafka"} 0.000198245
prometheus_target_sync_length_seconds_count{scrape_job="kafka"} 1

My guess is prometheous is not getting any metrics on port 7071; which aligns with earlier finding that JMX server is not respond on port 7071.

can you help me enabling kafka monitoring using JMX-exporter and Prometheus?

ankit patel
  • 1,399
  • 5
  • 17
  • 29
  • You downloaded `kafka-0-8-2.yml` but your command is using `/etc/jmx-exporter/kafka.yml`. Did you move/rename that file? – OneCricketeer Sep 20 '19 at 18:58
  • they are same files; i renamed after i copied them; i originally got the error indicating missing file but after renaming, this should not be the issue. i will update the question to avoid confusion thanks – ankit patel Sep 20 '19 at 19:06

1 Answers1

0

I have Prometheus running in a container

You need to configure Prometheus to scrape your external LAN IP, then because you're running Kafka outside of a container.

You can see on this line that the connection is being refused with your current setup

net_conntrack_dialer_conn_failed_total{dialer_name="kafka",reason="refused"} 79

You should either run Prometheus on your host and scrape localhost:7071

Or run Kafka in a container if you want kafka:7071 to be discoverable by Prometheus

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245