Metrics not visible in prometheus

Question

I am using elastic search exporter to pull elastic search metrics to prometheus. I installed the helm chart and could see the metrics on http://127.0.0.1:9108/metrics with port forwarding. But i don't see any metrics coming to prometheus. Can someone please tell me where to start troubleshooting the issue?

Answer 1

There are a few options that might help you:

• Check ServiceMonitor configuration of prometheus-elasticsearch-exporter and ensure that it is enabled:

serviceMonitor:
  enabled: true

• Read about Scrape configuration of kube-prometheus-stack and setup it according to your goals.

By default, Prometheus discovers PodMonitors and ServiceMonitors within its namespace, that are labeled with the same release tag as the prometheus-operator release. Sometimes, you may need to discover custom PodMonitors/ServiceMonitors, for example used to scrape data from third-party applications. An easy way of doing this, without compromising the default PodMonitors/ServiceMonitors discovery, is allowing Prometheus to discover all PodMonitors/ServiceMonitors within its namespace, without applying label filtering. To do so, you can set prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues and prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues to false.

Answer 2

TL;DR

You haven't setup PodMonitor or ServiceMonitor yet.
You need to create one according to prometheus-operator/Documentation#servicemonitor

---

You can use my helm chart kehao95/helm-prometheus-exporter to install various of prometheus exporters, including elasticsearch-exporter.
It will create not only the exporter Deployment but also Service/ServiceMonitor/PrometheusRule for you.

1. install chart

helm repo add kehao95 https://kehao95.github.io/helm-prometheus-exporter

2. get elasticsearch-exporter example

wget https://raw.githubusercontent.com/kehao95/helm-prometheus-exporter/master/examples/elasticsearch-exporter.values.yaml
# or create a new one
helm show values kehao95/prometheus-exporter > elasticsearch-exporter.values.yaml

3. update the elasticsearch connection info in the values file.
4. install with helm

helm install -n monitoring elasticsearch-exporter kehao95/prometheus-exporter -f elasticsearch-exporter.values.yaml

Explain

Since you could see the metrics via port-forward on you exporter. That means you have the exporter working. But the data will not come to prometheus automatically. You need to config your prometheus to scrape on the exporter's metrics endpoint.

A working stack of exporter in prometheus-operator requires:

• Deployment of the exporter (you have that already)

• Service selecting the exporter (depending on the Helm Chart you're using you may have that also)

• ServiceMonitor which declaratively specifies how groups of Kubernetes services should be monitored. The Operator automatically generates Prometheus scrape configuration based on the current state of the objects in the API server.

• PrometheusRule defines a desired set of Prometheus alerting and/or recording rules. The Operator generates a rule file, which can be used by Prometheus instances.