0

Hi All this question is in continuation of question here

by now i have single node kafka broker running on node x214 and its reporting metrics using jmx exporter on port 7071 

curl -s localhost:7071 | grep -i kafka
# reports many metrics on std out

i want to connect jmx exporter with Prometheus and eventually to grafana for visualization; as described here

i tried installing Prometheus as blogs explains but running into issues; i find launching Prometheus easy with docker container as 

docker run -p 9090:9090 -v /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

I am running Prometheus on node x215 (different node than kafka broker). i can reach to Prometheus gui on http://x215-ip:9090

prometheus.yaml file is set to following.

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus_master'
    scrape_interval: 5s
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['IP-215:9090']
  - job_name: ‘kafka-server’
    static_configs:
      - targets: [‘IP-214:7071’]

Issue :- on Prometheus gui i cant find any kafka metrics; as visible in response of curl command above

response of metrics page (IP-215:9090/metrics) is here

i feel that JMX exporter is reporting metrics correctly but its not connected to Prometheus correctly. can you help me here?

thanks

ankit patel
  • 1,399
  • 5
  • 17
  • 29
  • Is http://IP-214:7071/metrics accessible ? – Michael Doubez Oct 30 '19 at 22:32
  • no sorry i ll remove hyperlink – ankit patel Oct 30 '19 at 23:22
  • Rephrase the question. Can you shell into the container and curl that metrics endpoint? Do you need Prometheus in Docker on the other machine? Is the exporter only binding on localhost? Those are what we mean by accessible – OneCricketeer Oct 31 '19 at 00:08
  • Also, `curl -s localhost:7071` is different than `curl -s localhost:7071/metrics` – OneCricketeer Oct 31 '19 at 00:09
  • Repgrasing : Is http://IP-214:7071/metrics accessible from Prometheus ? – Michael Doubez Oct 31 '19 at 10:57
  • yes output is avaialbel at https://github.com/ankitpateledu/backup/blob/master/Prometheus_metrics_page.log – ankit patel Oct 31 '19 at 16:42
  • @cricket_007 i will check from within Prometheus docker container and see curl can get response back. from base machine x215 where docker container is running curl can get response from IP-215:7071. I have Prometheus working on base machine now; but i still need it in container to take it to my main cluster set up; so i will keep this question open – ankit patel Oct 31 '19 at 16:47
  • Well `IP-215` from machine `x215` should be expected to work because it would just go over the loopback interface anyway as with using `localhost` – OneCricketeer Oct 31 '19 at 17:13

1 Answers1

1

We need a little more information to help determine what's going on here. Can you share the following:

  • the bound interface for JMX exporter (is it listening on 0.0.0.0, 127.0.0.1, or some other specific interface?) This can be checked with something like ss -plunt | grep -i jmx
  • can the prometheus instance access the JMX Exporter instance over the network? Given that they're on different machines, there are numerous things that could come in the way (Firewalls, Network Configuration, SDN, IPtables... to name a few). The prometheus container does have telnet, so it's easiest to just use that to see if you can connect to port 7071. docker exec -it prometheus sh and run telnet IP-214:7071. You may also be able to curl, but curl is not available in the prometheus image.

Checking both of these can help rule out some simple problems in your configuration.

AgrahamLincoln
  • 130
  • 5