9

I am trying to scrape pod level info using prometheus kubernetes. Here is the config i am using:

 - job_name: 'kubernetes-pods'

  kubernetes_sd_configs:
  - api_servers:
    - 'https://kubernetes.default'
    role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: (.+):(?:\d+);(\d+)
    replacement: ${1}:${2}
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_pod_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

But i don't see any info on grafana. Do I need to make any changes in my apps? snapshot

Carl Bergquist
  • 3,894
  • 2
  • 25
  • 42
nocturnal
  • 395
  • 2
  • 6
  • 15
  • This article https://www.weave.works/blog/prometheus-and-kubernetes-monitoring-your-applications/ was very useful for me. – weivall Jul 25 '17 at 12:23

3 Answers3

7

With that configuration the first action asks that the pod be annotated with prometheus.io/scrape=true. Have you set that annotation on the pods in question?

brian-brazil
  • 31,678
  • 6
  • 93
  • 86
  • Yes i have set the annotations. Is there anything else i need to do? – nocturnal Jan 18 '17 at 18:55
  • From the pod (app) perspective, apart from setting annotations, do i need to publish any data or metrics? – nocturnal Jan 18 '17 at 19:16
  • Your applications need to expose data in the Prometheus format over HTTP (typically via a client library). Prometheus can't extract data from nothing. – brian-brazil Jan 18 '17 at 22:18
1

Be aware,

there is a configuration mistake here: __meta_kubernetes_pod_namespace does not exist and should be __meta_kubernetes_namespace

Best regards,

Bart

Boeboe
  • 2,070
  • 1
  • 17
  • 21
0

you should make sure that the data which your Grafana graphs use is actually present in Prometheus.

I assume you're using this default Grafana dashboard.

Your Grafana query might look like this:

sum (container_memory_working_set_bytes{pod_name=~"^$Pod$"}) / sum (machine_memory_bytes{kubernetes_io_hostname=~"^$Node$"}) * 100

So you'd have to make sure that the container_memory_working_set_bytes metric is recorded and that it has a field pod_name.

In case the metric is not recorded you'd have to do some digging in the logs of prometheus to check whether it is able to crawl the API.

In case the metric is there but the labels are wrong you could either adjust the query in Grafana or add another relabel_config in Prometheus. See official documentation.

pagid
  • 13,559
  • 11
  • 78
  • 104
  • I see that under http://x.x.x.x:9090/targets, kuberneted-pods, there is an error Get http://x.x.x.x:9100/metrics: dial tcp x.x.x.x:9100: getsockopt: connection refused. So its not even able to scrape data – nocturnal Jan 18 '17 at 19:48
  • Check the logs of your prometheus pod then and try to resolve possible errors. For me it was bad credentials as described here: https://github.com/kubernetes/dashboard/issues/374 – pagid Jan 18 '17 at 20:05
  • Did you find a solution for this? – pagid Feb 21 '17 at 21:42
  • As of now i am using cadvisor to get pod level info like memory usage and cpu usage. Also have implemented the app to expose some metrics which prometheus scrapes. – nocturnal Feb 28 '17 at 17:56
  • @nocturnal did you use annotations instead of labels? – atomaras Jun 06 '18 at 18:20