How to create a ServiceMonitor for prometheus-operator?

Question

Recently, prometheus-operator has been promoted to stable helm chart (https://github.com/helm/charts/tree/master/stable/prometheus-operator).

I'd like to understand how to add a custom application to monitoring by prometheus-operator in a k8s cluster. An example for say gitlab runner which by default provides metrics on 9252 would be appreciated (https://docs.gitlab.com/runner/monitoring/#configuration-of-the-metrics-http-server).

I have a rudimentary yaml that obviously doesn't work but also not provides any feedback on what isn't working:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: gitlab-monitor
  # Change this to the namespace the Prometheus instance is running in
  namespace: default
  labels:
    app: gitlab-runner-gitlab-runner
    release: prometheus
spec:
  selector:
    matchLabels:
      app: gitlab-runner-gitlab-runner
  namespaceSelector:
    # matchNames:
    # - default
    any: true
  endpoints:
  - port: http-metrics
    interval: 15s

This is the prometheus configuration:

> kubectl get prometheus -o yaml

...
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
  matchLabels:
    release: prometheus
...

So the selectors should match. By "not working" I mean that the endpoints do not appear in the prometheus UI.

Could you please post the output of `kubectl get prometheus -o yaml`. There you have to specify `serviceMonitorNamespaceSelector: {}` and `serviceMonitorSelector: {}` (example will select all serviceMonitors in all namespaces). Your `ServiceMonitor` definition looks fine to me. — Peter Ittner, Oct 25 '18 at 13:59
could you please define `serviceMonitorSelector: {}` just to exclude that this selector does not work properly. You can also check the annotations of your services. I have a `selector matchLabels: : ` in my case and it does work. I'm not sure if `namespaceSelector: # matchNames: # - default any: true ` also works. — Peter Ittner, Oct 26 '18 at 06:48
Based on the monitors shipped with prometheus operator I can even drop the namespaceSelector altogether (though I've tried both approaches). Not showing up in prometheus targets. Is there any logfile I could check for where service monitors are getting picked up? — andig, Oct 26 '18 at 14:19
Bang head against wall. Looking at https://github.com/coreos/prometheus-operator/blob/f9bc0aa0fd9aa936f500d9d241098863c60d873d/Documentation/user-guides/running-exporters.md I need a service too, not only a service monitor? — andig, Oct 26 '18 at 14:37

score 42 · Accepted Answer · answered Oct 26 '18 at 16:36

Thanks to Peter who showed me that it idea in principle wasn't entirely incorrect I've found the missing link. As a servicemonitor does monitor services (haha), I missed the part of creating a service which isn't part of the gitlab helm chart. Finally this yaml did the trick for me and the metrics appear in Prometheus:

# Service targeting gitlab instances
apiVersion: v1
kind: Service
metadata:
  name: gitlab-metrics
  labels:
    app: gitlab-runner-gitlab-runner
spec:
  ports:
  - name: metrics # expose metrics port
    port: 9252 # defined in gitlab chart
    targetPort: metrics
    protocol: TCP
  selector:
    app: gitlab-runner-gitlab-runner # target gitlab pods
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: gitlab-metrics-servicemonitor
  # Change this to the namespace the Prometheus instance is running in
  # namespace: default
  labels:
    app: gitlab-runner-gitlab-runner
    release: prometheus
spec:
  selector:
    matchLabels:
      app: gitlab-runner-gitlab-runner # target gitlab service
  endpoints:
  - port: metrics
    interval: 15s

Nice to know: the metrics targetPort is defined in the gitlab runner chart.

Use `release: kube-prometheus-stack` tag instead of `release: prometheus` when installed operator from kube-prometheus stack helm — Francisco Cardoso, Dec 22 '21 at 18:48
Very nice but what namespace should hold your ServiceMonitor gitlab-metrics-servicemonitor? — elmazzun, Oct 27 '22 at 13:59

score 31 · Answer 2 · answered Dec 22 '21 at 19:02

31

This image perfectly shows the connection between Prometheus,ServiceMonitors and Services

If any of the matches are not correct, targets won't show up.

Read more: https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/troubleshooting.md#troubleshooting-servicemonitor-changes

answered Dec 22 '21 at 19:02

Francisco Cardoso

1,438
15
20

Actually, Prometheus doesn't scrape metrics from the Service. Prometheus asks the Service "hey what are your Endpoints?". Then Prometheus will scrape each Endpoint individually (which is the same as scrapping each pod individually). Which is why usually every pod has an exporter side-car, so metrics can be reported per pod. – AFP_555 Jun 23 '22 at 16:41
1

I really like the image depicting the concept of ServiceMonitor, whenever a colleague asks how does it work I show them this picture. – Pavol Krajkovič Aug 16 '22 at 13:03

score 23 · Answer 3 · answered Jun 01 '19 at 11:42

23

I know this question is already answered. But I had a similar problem when Prometheus deployed in Kubernetes with Helm's stable/prometheus-operator chart couldn't find any active targets for my ServiceMonitor. It turned out that my Service exposed a port that I didn't explicitly named:

  - protocol: TCP
    port: 8080
    targetPort: uwsgi

I could use it in Ingress by targeting uwsgi port. But it seems that ServiceMonitor needs an explicitly named port in Service even if it has the same name as its own tagetPort:

  - name: uwsgi
    protocol: TCP
    port: 8080
    targetPort: uwsgi

I have written a blog post about this problem here

answered Jun 01 '19 at 11:42

vrs

1,922
16
23

1

What if I use a port directly, like 7896? I think it is okay too – 4t8dds Dec 19 '19 at 14:31
@vrs, i've tried your fix but didn't work. can you please help? https://stackoverflow.com/questions/60780448/setup-prometheus-operator-to-monitor-net-core-app – ShaneKm Mar 23 '20 at 14:26
Thanks for the blog. Although, now I am confused as to what purpose is served by the annotations `prometheus.io/scrape` and `prometheus.io/port` on `Deployments`? – cogitoergosum Jan 23 '22 at 07:51
Purpose is the same and should work as well, but annotations considered a deprecated way of targeting the services, AFAIK – Timur Bakeyev Aug 04 '22 at 21:54

score 2 · Answer 4 · answered Nov 22 '21 at 23:26

The above solutions are working fine till now.

The Release Label is important. Without this, Prom cannot add the app metrics to its target list.

Make sure you add the correct Release Label by checking in the ServiceMonitor of Prometheus itself. Also make sure to add the release Label to the Service and Deployment file too, in the metadata & Specs section.

If you encounter a situation where Prometheus is showing the Target but not the endpoints, take a look at this: https://github.com/prometheus-operator/prometheus-operator/issues/3053

This is how you find and check the serviceMonitor selector from Prometheus: `kubectl -n get prometheus` and then `kubectl -n get prometheus -oyaml `. Look for the attribute `serviceMonitorSelector:` — Francisco Cardoso, Dec 22 '21 at 18:58

How to create a ServiceMonitor for prometheus-operator?

4 Answers4

Linked