1

We have recently introduced Karpenter autoscaler in our Kubernetes infrastructure.

One of its strength is the deprovisioning mechanism.

However given that we have a lot of stateful workload, this can cause some serious issue at time. We are taking several measures to somewhat control it.

Meanwhile, we realize that we have no ways to observe how the measures we take mitigate the issue.

Hence the question: Is there way to monitor Kubernetes POD movements, i.e. tracking how many time a POD (not the containers) may have been re-scheduled / restarted to move from a node to another, etc....

Kins
  • 547
  • 1
  • 5
  • 22
MaatDeamon
  • 9,532
  • 9
  • 60
  • 127

1 Answers1

2

There is a metric in kubelet kube_pod_status_scheduled_time you could use. If the start time is more than scheduled time, it is scheduled:

kube_pod_start_time{namespace="my-ns"} - kube_pod_status_scheduled_time{namespace="my-ns"} > 0

You can also find the status of pods (Evicted, NodeAffinity, Shutdown ...) , with the metric kube_pod_status_reason.

Andromeda
  • 1,205
  • 1
  • 14
  • 21