Programmatically find overcommitted Kubernetes nodes

Question

I would like to be able to programmatically query Kubernetes to find overcommitted nodes.

If I do kubectl describe nodes, I get human-readable output including information about resource usage that I'm after, e.g.

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                   Requests              Limits
  --------                   --------              ------
  cpu                        786m (40%)            5078m (263%)
  memory                     8237973504500m (74%)  13742432Ki (126%)

However, kubectl describe doesn't support JSON or YAML output, and kubectl get nodes -ojson doesn't include the allocated resource stats. Is there any other way to access this information?

This is a duplication of : https://stackoverflow.com/questions/37464518/how-to-format-the-output-of-kubectl-describe-to-json — dzof31, Aug 28 '19 at 13:32
@dzof31 No, it is not a duplicate. This question concerns programatically finding overcommitted Kubernetes nodes, and there may be other approaches than through `kubectl describe`. Indeed, you'll notice that the accepted answer to the question you linked would not solve this problem. — Matt R, Aug 28 '19 at 13:45
@dzof31 except there is a specific API answer for getting the resources values for `describe nodes` — Matt, Aug 29 '19 at 01:34

Matt · Answer 1 · 2019-08-29T01:37:14.380

If you run any kubectl command with the --v=6 option the output will include the kubernetes API calls that make up the output.

In the case of kubectl describe nodes NODE you will see there is a api/v1/pods request that filters pods on the node and removes some "not running" statuses

I0828 13:44:29.310208   55233 round_trippers.go:438] GET https://kubernetes.docker.internal:6443/api/v1/pods?fieldSelector=spec.nodeName%3Ddocker-desktop%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded 200 OK in 4 milliseconds

If you complete this request with authentication information from your ~/.kube/config file you should be able to get the output. In this case, using jq to filter the output down to the resources component of the container spec with CA/Cert/Key auth (base64 decoded).

curl --cacert ~/.kube/docker-desktop.ca \
 --cert ~/.kube/docker-desktop.cert \
 --key ~/.kube/docker-desktop.key \
 https://kubernetes.docker.internal:6443/api/v1/pods?fieldSelector=spec.nodeName%3Ddocker-desktop%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded \
| jq '.items[].spec.containers[].resources'

{}
{}
{
  "limits": {
    "memory": "170Mi"
  },
  "requests": {
    "cpu": "100m",
    "memory": "70Mi"
  }
}
{
  "limits": {
    "memory": "170Mi"
  },
  "requests": {
    "cpu": "100m",
    "memory": "70Mi"
  }
}
{}
{
  "requests": {
    "cpu": "250m"
  }
}
{
  "requests": {
    "cpu": "200m"
  }
}
{}
{
  "requests": {
    "cpu": "100m"
  }
}
{}
{}

Running these calls and filters will generally be easier with one of the kubernetes API clients if you are regularly going to this level.

I should also note that `kubectl top` will give you instantaneous values for memory from nodes, if you have heapster or metrics-server installed and kubelet is setup correctly to use cgroups. This is collected from the current cgroups on the node so is the more the "actual usage" value for a node than what a node is committed to. — Matt, Aug 29 '19 at 01:30
I have an [answer over here](https://stackoverflow.com/a/56799354/1318694) that details the difference between the two types of memory tracking — Matt, Aug 29 '19 at 01:31

score 0 · Answer 2 · answered Aug 28 '19 at 13:17

0

You can write the output to a file maybe?

kubectl describe > output.txt

answered Aug 28 '19 at 13:17

Muhammad Ahsan

21
3

The output of `kubectl describe` is not suited for machine processing (not to say it cannot be done, but it would be hacky and brittle, and it would not be a good engineering choice unless there were no other option.) – Matt R Aug 28 '19 at 13:41

Programmatically find overcommitted Kubernetes nodes

2 Answers2