Pod in pending state due to Insufficient CPU

Question

On my GCE Kubernetes cluster I can no longer create pods.

Warning FailedScheduling    pod (www.caveconditions.com-f1be467e31c7b00bc983fbe5efdbb8eb-438ef) failed to fit in any node
fit failure on node (gke-prod-cluster-default-pool-b39c7f0c-c0ug): Insufficient CPU

Looking at the allocated stats of that node

Non-terminated Pods:        (8 in total)
  Namespace         Name                                        CPU Requests    CPU Limits  Memory Requests Memory Limits
  ---------         ----                                        ------------    ----------  --------------- -------------
  default           dev.caveconditions.com-n80z8                            100m (10%)  0 (0%)      0 (0%)      0 (0%)
  default           lamp-cnmrc                                  100m (10%)  0 (0%)      0 (0%)      0 (0%)
  default           mongo-2-h59ly                                   200m (20%)  0 (0%)      0 (0%)      0 (0%)
  default           www.caveconditions.com-tl7pa                            100m (10%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           fluentd-cloud-logging-gke-prod-cluster-default-pool-b39c7f0c-c0ug       100m (10%)  0 (0%)      200Mi (5%)  200Mi (5%)
  kube-system           kube-dns-v17-qp5la                              110m (11%)  110m (11%)  120Mi (3%)  220Mi (5%)
  kube-system           kube-proxy-gke-prod-cluster-default-pool-b39c7f0c-c0ug              100m (10%)  0 (0%)      0 (0%)      0 (0%)
  kube-system           kubernetes-dashboard-v1.1.0-orphh                       100m (10%)  100m (10%)  50Mi (1%)   50Mi (1%)
Allocated resources:
  (Total limits may be over 100%, i.e., overcommitted. More info: http://releases.k8s.io/HEAD/docs/user-guide/compute-resources.md)
  CPU Requests  CPU Limits  Memory Requests Memory Limits
  ------------  ----------  --------------- -------------
  910m (91%)    210m (21%)  370Mi (9%)  470Mi (12%)

Sure I have 91% allocated and can not fit another 10% into it. But is it not possible to over commit resources?

The usage of the server is at about 10% CPU average

What changes do I need to make for my Kubernetes cluster to be able to create more pods?

Answer 1

I recently had this same issue. After some research, I found that GKE has a default LimitRange with CPU requests limit set to 100m.

You can validate this by running kubectl get limitrange -o=yaml. It's going to display something like this:

apiVersion: v1
items:
- apiVersion: v1
  kind: LimitRange
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"v1","kind":"LimitRange","metadata":{"annotations":{},"name":"limits","namespace":"default"},"spec":{"limits":[{"defaultRequest":{"cpu":"100m"},"type":"Container"}]}}
    creationTimestamp: 2017-11-16T12:15:40Z
    name: limits
    namespace: default
    resourceVersion: "18741722"
    selfLink: /api/v1/namespaces/default/limitranges/limits
    uid: dcb25a24-cac7-11e7-a3d5-42010a8001b6
  spec:
    limits:
    - defaultRequest:
        cpu: 100m
      type: Container
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

This limit is applied to every container. So, for instance, if you have a 4 cores node and each pod creates 2 containers, it will allow only for around ~20 pods to be created (4 cpus = 4000m -> / 100m = 40 -> / 2 = 20).

The "fix" here is to change the default LimitRange to one that better fits your use-case and then remove old pods allowing them to be recreated with the updated values. Another (and probably better) option is to directly set the CPU limits on each deployment/pod definition you have.

Some reading material:

https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/#specify-a-cpu-request-and-a-cpu-limit

https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/cpu-default-namespace/#create-a-limitrange-and-a-pod

https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how-pods-with-resource-limits-are-run

https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-resource-requests-and-limits

Answer 2

I had the same issue when attempting to deploy to the cluster. In my case, there were unneeded pods being automatically created for test branches of my application. To diagnose the issue, I needed to do:

kubectl get po

kubectl describe po - for one of the existing pods, to check which node it's running on

kubectl get nodes

kubectl describe node - to see the CPU usage for the node being used by the existing pod, as below:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                       Requests      Limits
  --------                       --------      ------
  cpu                            1010m (93%)   4 (210%)

Then, the unneeded pods could be deleted using:

kubectl get deployments

kubectl delete deployment .... - then name of the deployment for the pod I needed to delete.

Once I deleted enough unused pods, I was ably to deploy new ones.

Answer 3

Yes, the overcommit is currently not supported. It's in planned improvements http://kubernetes.io/docs/user-guide/compute-resources. Related issue on github: https://github.com/kubernetes/kubernetes/issues/168

ps: in theory you can define custom node capacity, but I not sure.

Answer 4

TL;DR if you do use Limits and Requests: Lower the CPU requests and/or limits.

Context:

I was managing CPU and Memory pretty close to the limits in my case and here's what I found after some time.

Suppose that I have the following:

• A node named node-1, with CPU/A = 2000 millicpu and MEM/A = 7827 bytes
• An app named app-1 I want to deploy with Helm that takes all that's theoretically available.

See screenshot below, names are slightly censored for reasons but you see the big picture.

In my case, I configure app-1 with some resources.limits.cpu, resources.limits.memory, resources.requests.cpu, resources.requests.memory with resources.limits.cpu initially set to 1000m.

What exactly happens:

So node-1 not only runs app-1, but also 3 other extra apps for other things. From top to bottom, the sum of CPU requests is 100m + 0 + 250m + 1 (1000m) = 1350m (find it also in the allocation resources below).

All good right? Looks like it, but what if I want to deploy another version of app-1? In that case I would have to drop the old one and re-create it.

That might be acceptable in some cases, but when I want to deploy an update to app-1 with Helm in my case (helm update will delete the old Pod and start a new one mind you) without taking the old one down first then I would get the Insufficient CPU error.

That's because the kube-scheduler "might" be doing the following. It takes whatever previous value you have for CPU and add its to whatever limit the new app-1 you want to deploy has. Making the CPU requests theoretically go "beyond the requested limit". In other words it's doing an 1350 m + 1 (1000m) = 2350m for a very short time before the other Pod is brought down. Now this is the problem since that's beyond the initial limit of 2000m.

Solution:

The solution in this case is to set the CPU Request to a lower number, maybe only 500m so that the initial value is 100m + 0 + 250m + 500m = 850m, when it does the addition then it's only 100m + 0 + 250m + 500m + 500m = 1350m which is still below the hard limit of 2000m. Now in my case, helm will delete the old Pod bringing the whole Request CPU usage back to 850m but there is a very short span where it sums up the rest.

Answer 5

For me, creating all the deployments and services in a different namespace (other than default) fixed this issue.