8

I'm using this prometheus helm chart.

I was wondering if it is possible to setup the prometheus operator to automatically monitor every service in the cluster or namespace without having to create a ServiceMonitor for every service.

With the current setup, when I want to monitor a service, I have to create a ServiceMonitor with the label release: prometheus.

Edit:

Service with monitoring: "true" label

apiVersion: v1
kind: Service
metadata:
  name: issue-manager-service
  labels:
    app: issue-manager-app
    monitoring: "true"
spec:
  selector:
    app: issue-manager-app
  ports:
    - protocol: TCP
      name: http
      port: 80
      targetPort: 7200

"Catch-All" Servicemonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: service-monitor-scraper
  labels:
    release: prometheus
spec:
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
  jobLabel: monitoring
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      monitoring: "true"
Jonas
  • 7,089
  • 15
  • 49
  • 110
  • You have defined port `metrics` in your `serviceMonitor` snippet but you use `http` as a name in your `service`. Is it typo? – Rohlik Feb 24 '22 at 10:22

1 Answers1

14

Only if you have a common label on all services

# for example:
org: "my-company"
# or
monitoring: "true"
# or 
app.kubernetes.io/managed-by: "Helm"  # <- in most cases this represents all

Then you define a single, cross-namespace ServiceMonitor, that covers all labeled services:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: common-monitor
  namespace: monitoring
spec:
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
  jobLabel: monitoring
  namespaceSelector:
    any: true  # <- important
  selector:
    matchLabels:
      monitoring: "true"  # <- should match what you've chosen as common

Then to make sure this ServiceMonitor is discovered by the Prometheus Operator you either:

This additional explicit linkage between Prometheus Operator and ServiceMonitor is done intentionally - in this way, if you have 2 Prometheus instances on your cluster (say Infra and Product) you can separate which Prometheus will get which Pods to its scraping config.

From your question, it sounds like you already have a serviceMonitorSelector based on release: prometheus label - try adding that on your catch-all ServiceMonitor as well.

Max Lobur
  • 5,662
  • 22
  • 35
  • Thanks for the help. I have created the ServiceMonitor and only changed `monitoring: true` to `monitoring: "true" ` because I got an error otherwise. I have applied this file and then added the label `monitoring: "true"` to a service and applied it aswell. However the Service does not show up in my prometheus targets. Any Ideas? When should the ServiceMonitor pick up the Service with the `monitoring: "true"` label? – Jonas Oct 21 '20 at 08:49
  • Hold on, I added a description of the other part - how ServiceMonitor themselves are being discovered by an operator – Max Lobur Oct 21 '20 at 15:13
  • If everything is done correctly, the service is getting into a target list within a minute – Max Lobur Oct 21 '20 at 15:38
  • Hm I have added the `release: prometheus ` label to my "catch-all" servicemonitor. Now it shows up in the prometheus targets but when I add the label `monitoring: "true"` to a Service it still does not show up in the prometheus targets. I have added my yaml files to my question. Any idea what could be wrong? – Jonas Oct 22 '20 at 09:47
  • `port: metrics` - service port is called `http`. either change in servicemonitor, or service. It's the only issue I can see – Max Lobur Oct 22 '20 at 19:06
  • So I see in the documentation nothing about the ServiceMonitor now only annotations: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus#scraping-pod-metrics-via-annotations anyone knows if serviceMonitors are used anymore? – rufreakde Feb 08 '23 at 15:55