1

Problem

I want Prometheus to scrape a Java spring boot app in a K8s cluster. I have setup the config properly it seems, yet the UI page shows the service as "undefined":

enter image description here

Observations:

  1. Clicking on the undefined "show more" button in the UI shows __tmp_prometheus_job_name="serviceMonitor/default/java-server-prom-monitoring/0" .
  2. The Target list page does not have the my new service monitor at all, although the service discovery page does
  3. I confirmed that the configuration page includes updated information about the java server underneath scrape_configs:
scrape_configs:
- job_name: serviceMonitor/default/java-server-prom-monitoring/0
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /actuator/prometheus
  scheme: http
  follow_redirects: true
  enable_http2: true
  relabel_configs:
  - source_labels: [job]
    separator: ;
    regex: (.*)
    target_label: __tmp_prometheus_job_name
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
    separator: ;
    regex: (java-server);true
    replacement: $1
    action: keep
  1. I confirmed that the prometheus config file has prometheus as its release, and tried everything from this SO answer

Configuration File

apiVersion: v1
kind: Service
metadata:
  name: java-server-ip-service
spec:
  selector:
    app: java-server
  ports:
  - name: http-traffic
    protocol: TCP
    port: 8080
    targetPort: 8080
    
---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: java-server
  labels:
    release: prometheus
spec:
  replicas: 1 # number of replicas
  selector:
    matchLabels:
      app: java-server
  template:
    metadata:
      labels:
        app: java-server
    spec:
      containers:
        - name: java-server
          image: prometheusdemo/java-server # specify your react image and tag
          imagePullPolicy: Never
          ports:
            - containerPort: 8080 # container will listen here. should correspond to targetPort defined in service.
      
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: java-server-prom-monitoring
  labels:
    release: prometheus
  namespace: default
spec:
  selector:
    matchLabels:
      app: java-server
      release: prometheus
  endpoints:
    - port: http-traffic
      path: "/actuator/prometheus"

Info

  1. using the helm chart kube-prometheus-stack-45.20.0 from the prometheus-community repo
  2. using kubectl Client Version: v1.25.4

How can I get prometheus to connect to the java-server deployment?

J.E.C.
  • 2,616
  • 2
  • 18
  • 21
  • Are your deploying the `Service`|`Deployment` to the `default` Namespace? The `ServiceMonitor` explicitly specifies `namespace: default` but this value is unspecified|unset for `Service` and `Deployment`? Corrollary: could the `ServiceMonitor` in `default` be in a different namespace to `Service` and `Deployment`? – DazWilkin Apr 28 '23 at 01:11
  • Your `matchLabels` [`app: java-server`, `release: prometheus`] won't match either. All labels must match a `Service`'s labels and your `Service`` doesn't have any labels. – DazWilkin Apr 28 '23 at 01:18
  • @DazWilkin all resources are in `default`. What is your source for your second comment? The SO answer I referenced in my question body specifically matches with `release: prometheus`, and I've seen this on many other online pages. Are you saying that `matchLabels` should _only_ correspond to the `Service`? – J.E.C. Apr 28 '23 at 18:50
  • The `ServiceMonitor` selects `Service`'s. Your `matchLabels` selects `Service`'s with labels `app: java-server` and `release: prometheus`. The `Service` you included has no labels and so won't be selected. – DazWilkin Apr 28 '23 at 19:22

0 Answers0