2

I have installed Prometheus server and adapter on Kubernetes cluster, to collect metrics from Kubernetes objects.

Prometheus server and adapter are installed using Helm charts and they are running fine. Now deployed Spring boot app which serves metrics at the endpoint /actuator/prometheus.

added below annotation in deployment file:

annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/actuator/prometheus"

Now i could see custom metrics at pod level: enter image description here

When i try to see metrics at Service level, it is throwing error:

Error from server (NotFound): the server could not find the metric redis_queue_box_provider_size for service

metric samples collected into prometheus server: enter image description here

Is there any configuration i have to do to gather metrics at service level?

Rajeev
  • 4,762
  • 8
  • 41
  • 63
  • As far as I checked [here](https://sysdig.com/blog/kubernetes-monitoring-prometheus/), `Some services are designed to expose Prometheus metrics from the ground-up (the Kubernetes kubelet, Traefik web proxy, Istio microservice mesh, etc). Some other services are not natively integrated, but can be easily adapted using an exporter.` There is an example with redis in above link, and there is redis exporter [github](https://github.com/oliver006/redis_exporter). Could you try with that and let me know if that work? – Jakub Oct 06 '20 at 12:34
  • this is not external service. maybe the name misleading. treat it as custom metric. why this metric is not showing at service level – Rajeev Oct 06 '20 at 13:11
  • Yes, i though is was redis deployment because of the name. As mentioned in [prometheus values](https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L1340), `prometheus.io/scrape`: Only scrape services that have a value of `true`, you have that in your deployment, but did you add it in your service annotations? – Jakub Oct 07 '20 at 06:22
  • same behavior even i add at service level – Rajeev Oct 09 '20 at 11:45
  • 2
    What do you want to achieve? The linked piece of configuration by Jakub instructs to Prometheus to access all endpoints that belong to a service. That's hardly different from the pod-based discovery, except how the IP address of the pods are collected - you'll get one target per pod eventually. A service-based discovery is possible, meaning you want a single target per service, with the IP address of the load-balanced service ClusterIP. However, such a configuration is not provided by the Helm chart, you have to manually add this to the Prometheus config file (you have to use 'role: service') – Laszlo Valko Oct 11 '20 at 05:29

0 Answers0