2

Currently I am using Kubernetes v1.11.6. I deployed kubernetes in AWS by using KOPS. In k8s cluster, deployed kafka, elasticsearch.

PVC for kafka and elasticsearch are EBS volumes in AWS.

My question is how to monitor PVC used and remaining available.

This did not worked, How to monitor disk usage of kubernetes persistent volumes? They no longer seem to be exposed starting from 1.12

I thought of using aws cloudwatch but I am thinking kubernetes will have some answer for this generic problem.

I should be able to see PVC used and remaining available disk space

vertrost
  • 40
  • 6
Ram
  • 157
  • 1
  • 3
  • 8

1 Answers1

4

generally speaking you can monitor following metrics:

kubelet_volume_stats_capacity_bytes
kubelet_volume_stats_available_bytes

These metrics can be scraped from the kubelet endpoint on each node with tools like Prometheus :)

Stepan Vrany
  • 1,026
  • 9
  • 14
  • Thanks Stepan. I do not see kublet_* in prometheus. – Ram Oct 25 '19 at 20:57
  • Great, so you are using Prometheus. Are you using Prometheus operator (https://github.com/helm/charts/tree/master/stable/prometheus-operator) or just Prometheus (https://github.com/helm/charts/tree/master/stable/prometheus)? Which endpoints are you scraping? – Stepan Vrany Oct 26 '19 at 17:18
  • I followed this https://github.com/Thakurvaibhav/k8s/tree/master/monitoring/prometheus can you please check and let me know – Ram Oct 26 '19 at 17:50
  • It seems you are scraping only cAdvisor. Could you try to extend Prometheus config with `kubernetes-nodes` from this example? https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L53 – Stepan Vrany Oct 27 '19 at 10:41
  • Thanks Stepan, I will try. Could you please let me know how to visualize output, any grafana dashboard? – Ram Oct 29 '19 at 02:29
  • Promql will be something like `kubelet_volume_stats_available_bytes / kubelet_volume_stats_capacity_bytes * 100` I guess the Grafana expression will be almost the same. – Stepan Vrany Oct 29 '19 at 09:25
  • It worked ThumbsUp. Thanks Stepan. Only concern is I have some AWS EFS PVC's i.e created PVC from AWS EFS and EFS PVC's are not showing up in graph. how to scrape that EFS mounts disk usage? – Ram Oct 30 '19 at 02:41
  • 1
    EFS is not showing as disk when I mounted. drwxrws--x 6 root 2002 6144 Oct 30 01:00 prometheuspvc – Ram Oct 30 '19 at 02:58
  • @Ram Did you find any way to scrape the EFS PVC's disk usage? – Hemabh Jul 13 '22 at 06:55