21

I have container_fs_usage_bytes with prometheus to monitor container root fs, but it seems that there is no metrics for other volumes in cAdvisor.

hiroshi
  • 6,871
  • 3
  • 46
  • 59
  • 1
    There is a ticket in the community to work on the PVC disk usage data, here is the link, fyi: https://github.com/kubernetes/features/issues/293 – Tom Xing Jun 28 '17 at 07:59

3 Answers3

39

I confirmed that Kubernetes 1.8 expose metrics for prometheus.

  • kubelet_volume_stats_available_bytes
  • kubelet_volume_stats_capacity_bytes
  • kubelet_volume_stats_inodes
  • kubelet_volume_stats_inodes_free
  • kubelet_volume_stats_inodes_used
  • kubelet_volume_stats_used_bytes
hiroshi
  • 6,871
  • 3
  • 46
  • 59
7

Metrics for volumes are available via the kubelet summary API (/stats/summary). However, each volume plugin has to implement their own metrics. As of Kubernetes 1.7, the current volume plugins that have implemented metrics include: emptydir, secrets, gce pd, aws ebs, azure file, flocker, and portworx

Michelle
  • 331
  • 1
  • 3
  • I don't understand what this means yet. I'll find out /stats/summary API. I'm using GKE and gce pd. Will it be available through cAdvisor and prometheus after they rolling out kubernetes 1.7 to GKE? – hiroshi Jun 29 '17 at 00:23
6

The following metrics must be used for monitoring persistent volume stats in Kubernetes (the PVC name is exported in persistentvolumeclaim label):

  • kubelet_volume_stats_capacity_bytes - the per-PVC capacity in bytes.
  • kubelet_volume_stats_used_bytes - the per-PVC space usage in bytes.
  • kubelet_volume_stats_available_bytes - the per-PVC free space in bytes.

The following PromQL queries can be used for determining per-pod PVC disk space usage in bytes:

sum(kubelet_volume_stats_used_bytes) by (namespace,persistentvolumeclaim)
  * on(namespace,persistentvolumeclaim) group_left(pod)
kube_pod_spec_volumes_persistentvolumeclaims_info

The following query can be used for determining free PVC disk space in bytes per each pod:

sum(kubelet_volume_stats_available_bytes) by (namespace,persistentvolumeclaim)
  * on(namespace,persistentvolumeclaim) group_left(pod)
kube_pod_spec_volumes_persistentvolumeclaims_info
valyala
  • 11,669
  • 1
  • 59
  • 62
  • In my case, `kubelet_volume_stats_used_bytes` only contains the metrics for PVCs created with the default storageclass (gp2). I have other PVCs attached to an efs storageclass as well. How can I view their metrics? – Hemabh Jul 13 '22 at 06:48
  • [EFS](https://aws.amazon.com/efs/) has no upper size limits, e.g. the `kubelet_volume_stats_available_bytes` metric cannot be defined for EFS. Probably, that's why other `kubelet_volume_stats_*` metrics aren't exported for EFS volumes. – valyala Jul 13 '22 at 15:57
  • 1
    Thank you so much! I just to mention that `kubelet_` metrics are exported by kubelet ^^ and `kube_pod_` metrics are exported by kube-state-metrics. If using kube-prometheus-stack helm chart, both components can be easily enabled. – Nour Wolf Nov 06 '22 at 09:32