5

I have deployed application on kubernetes cluster and for monitoring using prometheus and grafana. For kubernetes pods monitoring using Grafana dashboard: Kubernetes cluster monitoring (via Prometheus) https://grafana.com/grafana/dashboards/315

I had imported the dashboard using id 315 and its reflecting without pod name and containers name instead getting pod_name . Can anyone pls help how can i get pod name and container name in dashboard.

enter image description here

enter image description here

enter image description here

muku
  • 91
  • 1
  • 7
  • Could you provide: 1. Screen of Settings > Variables 2.What version of Kubernetes are you using? I see that article was updated 2 yaers ago. 3. Your env is On-Perm or local? 4. Could you add your metrics query? – PjoterS Jan 07 '20 at 10:50
  • @PjoterS: Added screen setting of variable and kubernetes version is v1.17.0. This is my local environment project. This is not metrics query i have imported dashboard using *https://grafana.com/grafana/dashboards/315* – muku Jan 07 '20 at 12:13
  • thanks it works for me in Kubernetes 1.16 metrics labels like pod_name and container_name was removed. Instead of that you need to use pod and container. You can verify it here. – Ramakant Chandrakar Mar 12 '20 at 13:51

2 Answers2

9

Provided tutorial was updated 2 years ago.

Current version of Kubernetes is 1.17. As per tags, tutorial was tested on Prometheus v. 1.3.0, Kubernetes v.1.4.0 and Grafana v.3.1.1 which are quite old at the moment.

In requirements you have statement:

Prometheus will use metrics provided by cAdvisor via kubelet service (runs on each node of Kubernetes cluster by default) and via kube-apiserver service only.

In Kubernetes 1.16 metrics labels like pod_name and container_name was removed. Instead of that you need to use pod and container. You can verify it here.

Any Prometheus queries that match pod_name and container_name labels (e.g. cadvisor or kubelet probe metrics) must be updated to use pod and container instead.

Please check this Github Thread about dashboard bug for more information.

Solution

Please change pod_name to pod in your query.

PjoterS
  • 12,841
  • 1
  • 22
  • 54
  • No where i have manually mentioned the pod_name or container_name. I have imported the dashboard using https://grafana.com/grafana/dashboards/315. Can you pls help what need to be done. thanks – muku Jan 07 '20 at 12:24
  • You are using K8s 1.17 which removed `pod_name`. In your variables you have `$pod_name` with labal_values(..... ,`pod_name`). You need change `pod_name` to `pod` in `JSON Model`. If you are using somewhere in this dashboard `container_name` you also need to change it to `container`. – PjoterS Jan 07 '20 at 12:27
  • I have downloaded the json file of https://grafana.com/grafana/dashboards/1621 and replaced the pod_name to pod and container_name to container in "expr" and "legendFormat" but still not working. – muku Jan 08 '20 at 05:47
  • "expr": "sum (rate (container_cpu_usage_seconds_total{image!=\"\",name=~\"^k8s_.*\",kubernetes_io_hostname=~\"^$Node$\"}[1m])) by (pod)", "interval": "10s", "intervalFactor": 1, "legendFormat": "{{ pod }}", "metric": "container_cpu", "refId": "A", "step": 10 – muku Jan 08 '20 at 06:35
  • As I mentioned before, you used old version of Grafana, old Json file (last update 3 years ago.) and Kubernetes 1.17 which I dont think they are compatible. Please download latest version of Grafana 6.5.3 – PjoterS Jan 24 '20 at 11:12
2

Kubernetes version v1.16.0 has Removed cadvisor metric labels pod_name and container_name to match instrumentation guidelines. Any Prometheus queries that match pod_name and container_name labels (e.g. cadvisor or kubelet probe metrics) must be updated to use pod and container instead.

You can check: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.16.md#metrics-changes

mGeek
  • 786
  • 8
  • 11