2

I am using 'crictl' tool to work with containerd runtime containers (under kubernetes) in a managed cluster.

I'm trying to set the memory limit (in bytes) to 16MB with the command: crictl -r unix:///run/containerd/containerd.sock update --memory 16777216 c60df9ef3381e

And get the following error:

E1219 11:10:11.616194    1241 remote_runtime.go:640] "UpdateContainerResources from runtime service failed" err=<
    rpc error: code = Unknown desc = failed to update resources: failed to update resources: /usr/bin/runc did not terminate successfully: exit status 1: unable to set memory limit to 16777216 (current usage: 97058816, peak usage: 126517248)
    : unknown
 > containerID="c60df9ef3381e"
FATA[0000] updating container resources for "c60df9ef3381e": rpc error: code = Unknown desc = failed to update resources: failed to update resources: /usr/bin/runc did not terminate successfully: exit status 1: unable to set memory limit to 16777216 (current usage: 97058816, peak usage: 126517248)
: unknown

At first I thought that maybe I cannot set a memory limit directly to a running container lower than the limit that appears in the kubernetes yaml. Here Are the limits from K8s: Requests:{"cpu":"100m","memory":"64Mi"} Limits:{"cpu":"200m","memory":"128Mi"} But not, even setting a memory limit above the K8S request (e.g. 65MB) gives this same error!

This works on Docker runtime - I'm able to limit the memory of the container. Yes, it might crash, but the operation works..

Then, I tried to give a memory limit higher than the current usage, and it succeeded...

Can anyone help understanding this error and what might be causing it on containerd runtime?? Is this indeed a limitation that I cannot limit to a lower memory currently used by the container? Is there a way to overcome that?

Thanks a lot for your time!!!

Scorpicos
  • 23
  • 3
  • 1
    You can't make this change in the way you're describing without administrator-level login access to the Kubernetes nodes, which I wouldn't ordinarily expect a developer to have. In a Kubernetes context I'd expect to set `resources: { limits: }` in a Pod spec and not think about these lower-level details. – David Maze Dec 19 '22 at 14:49
  • Thanks @DavidMaze let me please clarify. You are right in a production environment, but in my setup (where i'm the admin) I'm doing some tests and want to check the impact of memory limit on containers and pods – Scorpicos Dec 20 '22 at 07:50

1 Answers1

1

The error suggests that the container is already using more memory than the limit you're trying to impose. If the limit were to be imposed in this case, then any further attempts to allocate memory would fail. That would usually cause processes to abort, which is bad. Also, the container would already be violating the limit you just set.

The limit isn't allowed to be imposed in this case because data loss is highly likely to occur, and the choice of which data to lose should not be left to chance.

I suggest if you want to lower the limit, you need to lower the current usage to below that limit before you do so, to retain control of what memory is freed.

Bart
  • 46
  • 4