Kubernetes Pod OOMKilled Issue

Question

The scenario is we run some web sites based on an nginx image in kubernetes cluster. When we had our cluster setup with nodes of 2cores and 4GB RAM each. The pods had the following configurations, cpu: 40m and memory: 100MiB. Later, we upgraded our cluster with nodes of 4cores and 8GB RAM each. But kept on getting OOMKilled in every pod. So we increased memory on every pods to around 300MiB and then every thing seems to be working fine.

My question is why does this happen and how do I solve it. P.S. if we revert back to each node being 2cores and 4GB RAM, the pods work just fine with decreased resources of 100MiB.

the logs -f command only reveals memory issue and that typical nginx error of child process being and nothing else :( — Shahnewaz Ul Islam Chowdhury, May 19 '20 at 10:40
Are you using an Autoscaler (HPA)? Is there any chance that the load/traffic for those pods increased? For example the traffic on your website increased and caused higher than usual load which caused OOMKiller to trigger. — Wytrzymały Wiktor, May 19 '20 at 10:50
not at all, just launching the pods with 100MiB in 4core nodes makes the pods to die with 00MKilled error so embarrassing the situation is right now :( — Shahnewaz Ul Islam Chowdhury, May 19 '20 at 12:33
Could you please share you pod yaml? Pod should not request more memory just because your node has more of it. I need to make sure the config is right. — Wytrzymały Wiktor, May 20 '20 at 09:29

score 0 · Answer 1 · answered May 27 '20 at 09:32

First of all, the pod should not request more memory/CPU just because the node has gotten more resources. Without your specs it is hard to point out what might be wrong config wise but I would like to explain the concept to make it more clear.

You mentioned your pod configuration but did not specify if these are limits or requests.

Requests are what the container is guaranteed to get. If a container requests a resource, Kubernetes will only schedule it on a node that can give it that resource. These will not cause OOMs, they will cause pod not to get scheduled.
Limits, on the other hand, make sure a container never goes above a certain value. This can cause OOM kill.

Requests and limits are on a per-container basis. While Pods usually contain a single container, it’s common to see Pods with multiple containers as well. Each container in the Pod gets its own individual limit and request, but because Pods are always scheduled as a group, you need to add the limits and requests for each container together to get an aggregate value for the Pod.

Below is an example of a pod that has 2 containers with the same specified requests and limits:

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: db
    image: mysql
    env:
    - name: MYSQL_ROOT_PASSWORD
      value: "password"
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  - name: wp
    image: wordpress
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

If you want to calculate requests and limits for the whole pod you need to sum those values, making it: request of 0.5 cpu and 128 MiB of memory, and a limit of 1 cpu and 256MiB of memory.

If you want to find more regarding that topic than check out the official documentation:

Please, let me now if that helped.

@ShahnewazUlIslamChowdhury Have you managed to make it work? — Wytrzymały Wiktor, Jun 01 '20 at 07:59

Kubernetes Pod OOMKilled Issue

1 Answers1