0

I have been trying to setup monitoring for a server which is on client side (unreachable).

One way I tried was prometheus remote write. As I am new to prometheus, I expected that Client prometheus will push the metrics to central prometheus further I can create a Grafana dashboard. I guess I am wrong, somehow I am getting this error:

"Failed to send batch, retrying" err="Post "http://xx.xx.xx.xx:9090/api/v1/write": context deadline exceeded"

I tried everything to solve this problem but nothing worked. Is it because both client and server prometheus are unreachable to each other? Is it necessary even in remote write config for prometheus to reach the endpoint? Any input is welcomed I am stuck for over months now.

UPDATE: I tried telegraf and influxdb instead of central prometheus this time both client prometheus and telegraf can ping eachother still I am getting the same error:

"Failed to send batch, retrying" err="Post "http://xx.xx.xx.xx:1234/receive": context deadline exceeded"

Uzmasaman Chanderki
  • 21
  • 1
  • 4

1 Answers1

0

Prometheus by default doesn't accept data via remote_write protocol. This option can be enabled by running a Prometheus with --enable-feature=remote-write-receiver command-line flag. See these docs.

Side notes:

  • You can also write the collected data from client-side Prometheus to any other supported centralized Prometheus-compatible remote storage from this list. Some of these systems support Prometheus query API, so they can be used as a drop-in replacement for Prometheus in Grafana. See, for example, the system I work on - VictoriaMetrics.

  • There are also lightweight alternatives to Prometheus, which can be used at client side in order to reduce resource usage: Prometheus agent and vmagent.

valyala
  • 11,669
  • 1
  • 59
  • 62
  • Hello, Thanks for the input. Yes, I have enable remote write flag on the central prometheus and client prometheus has remote write url for sending the metrics to central prometheus. Metrics are not sent to central prometheus. I am not able to figure out why. – Uzmasaman Chanderki Dec 22 '21 at 21:53