1

We are running Kubernetes with the following configuration: On-premise Kubernetes 1.11.3, cri-o 1.11.6 and CentOS7 with UEK-4.14.35

I can't make crictl stats to return pods information, it returns empty list only. Has anyone run into the same problem?

Another issue we have, is that when I query the kubelet for stats/summary it returns an empty pods list.

I think that these two issues are related, although I am not sure which one of them is the problem.

Dan Kohn
  • 33,811
  • 9
  • 84
  • 100
alubin
  • 11
  • 1
  • can you run pods? just an issue with stats? – Rico Oct 06 '18 at 18:40
  • Yes all is running well, it's just the stats, and a long with the kubelet means that HPA doesn't work – alubin Oct 08 '18 at 07:26
  • Have you checked `crictl info` and verify `CRI plugin` runtime status? How did you retrieve stats/summary from `kubelet`? – Nick_Kh Oct 08 '18 at 14:16
  • Yes, all is running in terms of cri-o pods are starting and terminating, happy campers. For the kubelet, I run, curl localhost:10255/stats/summary and the pods list is an empty array. The kubelet does provide general statistics, but not for pods – alubin Oct 10 '18 at 06:04
  • # crictl info { "status": { "conditions": [ { "type": "RuntimeReady", "status": true, "reason": "", "message": "" }, { "type": "NetworkReady", "status": true, "reason": "", "message": "" } ] } } – alubin Oct 10 '18 at 06:07

1 Answers1

1

I would recommend checking kubelet service to verify health status and debug any suspicious events within the cluster. I assume that CRI-O runtime engine can select kubelet as the main Pods information provider because of its managing Pod lifecycle role.

systemctl status kubelet -l

journalctl -u kubelet

In case you found some errors or dubious events, share it in a comment below this answer.

However, you can use metrics-server, which will collect Pod metrics in the cluster and enable kube-apiserver flags for Aggregation Layer. Here is a good article about Horizontal Pod Autoscaling and monitoring resources via Prometheus.

Nick_Kh
  • 5,089
  • 2
  • 10
  • 16