Pod CPU / Memory requests in error state are not released

Question

I have some pods created by CronJobs that are in Error state and it seems that the CPU / Memory requested by these pods are not released since kubelet is not killing them. It prevents other pods from being scheduled.

Is that the expected behavior ? Should I clean them by hand to get the resources back ?

Thanks.

Yes, if the pods were scheduled to the node and an error occurred after that, you need to delete those pods to cleanup resources. — Kamol Hasan, Dec 02 '20 at 11:16
Thank you for your reply @KamolHasan. What about pods in `Completed` state ? — cyberhippo, Dec 02 '20 at 13:06

score 1 · Accepted Answer · answered Dec 02 '20 at 13:18

Pods in Error you should delete this pod to release resources assigned to it.

However pods in Completed or Failed status are not required to be cleaned up to release resources allocated to them. It can be checked by running simple Job to check memory resources allocated in node.

Memory allocation before job:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                   Requests     Limits
  --------                   --------     ------
  cpu                        811m (86%)   1143m (121%)
  memory                     555Mi (19%)  1115Mi (39%)

Job example:

apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: null
  name: test-job
spec:
  template:
    metadata:
      creationTimestamp: null
    spec:
      containers:
      - command:
        - date
        image: busybox
        name: test-job
        resources:
          requests:
            memory: 200Mi
      restartPolicy: Never

Memory after the job deployment:

  Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                   Requests     Limits
  --------                   --------     ------
  cpu                        811m (86%)   1143m (121%)
  memory                     555Mi (19%)  1115Mi (39%)

Pod CPU / Memory requests in error state are not released

1 Answers1