How do you add scrape targets to a Prometheus server that was installed with Kubernetes-Helm?

Question

Background

I have installed Prometheus on my Kubernetes cluster (hosted on Google Container Engineer) using the Helm chart for Prometheus.

The Problem

I cannot figure out how to add scrape targets to the Prometheus server. The prometheus.io site describes how I can mount a prometheus.yml file (which contains a list of scrape targets) to a Prometheus Docker container -- I have done this locally and it works. However, I don't know how to specify scrape targets for a Prometheus setup installed via Kubernetes-Helm. Do I need to add a volume to the Prometheus server pod that contains the scrape targets, and therefore update the YAML files generated by Helm??

I am also not clear on how to expose metrics in a Kubernetes Pod -- do I need to forward a particular port?

score 24 · Accepted Answer · edited Jul 12 '18 at 14:36

24

You need to add annotations to the service you want to monitor.

apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/scrape: 'true'

From the prometheus.yml in the chart:

prometheus.io/scrape: Only scrape services that have a value of true
prometheus.io/scheme: http or https
prometheus.io/path: override if the metrics path is not /metrics
prometheus.io/port: If the metrics are exposed on a different port

And yes you need to expose the port with metrics to the service so Prometheus could access it

edited Jul 12 '18 at 14:36

BMW

42,880
12
99
116

answered Aug 12 '17 at 04:22

lwolf

960
9
16

Pls forgive my ignorance, but where did you find the prometheus.yml file? I can't find anything on the GitHub page for the chart. Also, I can forward a container port to the service, but is there a specific port to expose? And how do I point Prometheus to this port? – Ismail Khan Aug 12 '17 at 07:53
It's in the chart values file - https://github.com/kubernetes/charts/blob/master/stable/prometheus/values.yaml#L619 Regarding the port. If your service exposes metrics, it's either uses separate port (sidecar pattern) or the same as your application. What exactly are you trying to monitor? – lwolf Aug 12 '17 at 08:04
1

Ah I see. Seems like I don't have to specify a port if I just expose metrics on the same port as the service itself. – Ismail Khan Aug 12 '17 at 08:17
2

Is that everything? When should prometheus recognize the new service? I added the annotations to my service but prometheus does not register it. – Jonas Oct 21 '20 at 14:16
it is enough if you deploy prometheus using default helm chart. Service should appear on http://your-prometheus-url/targets – lwolf Oct 22 '20 at 16:36
I think this only works if we have Prometheus on same namespace – Ayush P Gupta Nov 24 '21 at 06:48
This is cool but where is this documented, I searched for these annotation and I can only find them in various blogs and tutorials, not on any official site. – Dojo Sep 28 '22 at 09:52

score 3 · Answer 2 · edited Nov 28 '21 at 10:55

First of all, you need to create a Service Monitor, which is a custom K8s resource. Just create a servicemonitor.yaml in the manifests folder.

Since when we are deploying on K8s, we don't have access to the Prometheus.yaml file to mention the targets, we create the servicemonitor, which in-turn adds the target to the scrap_config in the Prometheus.yaml file. You can read about it more from here.

This is a sample servicemonitor.yaml file for exposing Flask App metrics in Prometheus.

apiVersion: monitoring.coreos.com/v1 
kind: ServiceMonitor 
metadata:
  name: flask-metrics
  namespace: prometheus # namespace where prometheus is running
  labels:
    app: flask-app
    release: prom  # name of the release 
    # ( VERY IMPORTANT: You need to know the correct release name by viewing 
    # the servicemonitor of Prometheus itself: Without the correct name, 
    #  Prometheus cannot identify the metrics of the Flask app as the target.)
spec:
  selector:
    matchLabels:
      # Target app service
      app: flask-app # same as above
      release: prom # same as above
  endpoints:
  - interval: 15s # scrape interval
    path: /metrics # path to scrape
    port: http # named port in target app
  namespaceSelector:
    matchNames:
    - flask # namespace where the app is running

Also add this Release Label to the Services and Deployments file too, in the metadata and spec section.

If you encounter a situation where Prometheus is showing the Target but not the endpoints, take a look at this: https://github.com/prometheus-operator/prometheus-operator/issues/3053

Some useful links:

How do you add scrape targets to a Prometheus server that was installed with Kubernetes-Helm?

2 Answers2