Prometheus - getsockopt: connection refused

Question

In the targets page of Prometheus, I'm getting the following error:

I'm using it in Linux host

Prometheus Version:

prometheus, version 1.1.2 (branch: master, revision: 36fbdcc30fd13ad796381dc934742c559feeb1b5)
  build user:       root@a74d279a0d22
  build date:       20160908-13:12:43
  go version:       go1.6.3

What is the issue here?

score 4 · Answer 1 · answered Sep 19 '16 at 10:41

4

As the scrape error message says, Prometheus can't open a connection to your target at 172.19.36.189:9104. Can you connect to that IP/port manually? Are you perhaps running Prometheus in a container where it cannot reach that IP?

answered Sep 19 '16 at 10:41

Julius Volz

688
4
6

how to connect directly? I have one linux host and I have MySQL running in that. I have prometheus and grafana running in the same host. Connection is local only. Manual how? Do help Julius! – tesla747 Sep 19 '16 at 10:44
Julius can you help me out here? – tesla747 Sep 19 '16 at 13:46
@tesla747: Try wget or a browser: the endpoint also exposes an html page. – Martin Schröder Sep 19 '16 at 18:34
Yeah, try reaching http://172.19.36.189:9104/metrics in your browser. The problem is that Prometheus cannot open a TCP connection to that IP and port combination. That can either be because there really is nothing listening on that IP and port or because you have some setup in which a connection is prevented (running Prometheus in a container without network access to that IP, or running into other networking/firewall issues). At that point, it's not really a Prometheus-related question anymore. – Julius Volz Sep 20 '16 at 11:38

droidbot · Answer 2 · 2020-04-28T10:42:01.180

2

The connection may be failing anywhere in the traffic flow. This can help identify:

That said, if your service is running in the same host as Prometheus, you should try and use 127.0.0.1:9104/metrics as the endpoint in Prometheus scrape config, ensuring the service is indeed listening in port 9104.

As you try to troubleshoot and fix issues in each layer, check connectivity from your Prometheus host using a simple:

nc -v <service-ip> <service-port>

edited Apr 28 '20 at 10:42

answered Apr 28 '20 at 10:33

droidbot

937
1
10
26

I know it is an old question. Answered it to help anybody thats looking for similar problem. – droidbot Apr 28 '20 at 10:46
Does it make a difference to use `127.0.0.1:9104` instead of `localhost:9104`? – Danny Bullis Aug 17 '21 at 00:43

score -1 · Answer 3 · answered Feb 10 '18 at 04:41

Three things to check here:

Networking:

Is the scrape target http://172.19.36.189:9104/metrics reachable from where you've opened the Prometheus GUI in the browser? Check curl -vvv http://172.19.36.189:9104/metrics and any proxy to http://172.19.36.189:9104/metrics.

Prometheus logs:

Start prometheus with debugging turned on by using the flag --log.level:

$ /bin/prometheus -h
...
--log.level=info      [debug,
                      info,
                      warn,
                      error]

Then, check the container's logs using:

docker logs <name of prometheus container>
kubectl logs <name of prometheus pod>

Is the scrape target's port 9104 exposed in the container/pod/service?

Prometheus - getsockopt: connection refused

3 Answers3