33

How do I set ulimit for containers in Kubernetes? (specifically ulimit -u)

shaylevi2
  • 644
  • 1
  • 6
  • 14

5 Answers5

11

It appears that you can't currently set a ulimit but it is an open issue: https://github.com/kubernetes/kubernetes/issues/3595

James Brown
  • 658
  • 6
  • 13
5

If you are able to ssh into the kubernetes cluster, you can modify the docker.service file.

  • For an amazon EKS cluster, the file is located at /usr/lib/systemd/system/docker.service.

  • Append the property LimitMEMLOCK=Infinity in the file and then restart the docker service.

    sudo service docker restart

This would spin up docker containers with an infinite memlock value. Probably the equivalent command with docker cli is:

docker run --ulimit memlock=-1:-1 <docker image>

trex
  • 325
  • 3
  • 8
nikoo28
  • 2,961
  • 1
  • 29
  • 39
  • for containerd users without using dockershim : 1. sudo systemctl edit containerd 2. add this text [Service]LimitMEMLOCK=infinity 3. sudo systemctl daemon-reload && sudo systemctl restart containerd – ws_ Sep 29 '21 at 07:03
4

In Kubernetes cluster (AWS EKS) you can change the ulimit for a docker container by modifying the /etc/docker/daemon.json in the node where your container is running.

Add following lines to /etc/docker/daemon.json

"default-ulimits": { "nofile": { "Name": "nofile", "Hard": 128000, "Soft": 128000 } }

and finally restart the docker service on that node by executing following command.

service docker restart

Sachan
  • 91
  • 1
  • 1
    that doesn't sound like a sustainable practice, is there a way to set this on eks but have it carry forward for new nodes? – user3505901 Mar 10 '22 at 18:40
3

Above all not working for me.

I done the following (it works on ubuntu:18.04 and centos/7):

sudo nano /usr/lib/systemd/system/docker.service

Added

--default-ulimit memlock=-1:-1

To line

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

This line must looks like:

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --default-ulimit memlock=-1:-1

And then you MUST reload rightly: firstly run command

sudo systemctl daemon-reload

And then run command

sudo systemctl restart docker.service

To check work it or not works, run command

docker run busybox:1.28 cat /proc/1/limits

You must see unlimited max lock memory like about this:

...
Max locked memory         unlimited            unlimited            bytes
...

And elasticsearch starts to work!!!!

Евгений Колпаков
  • 96
  • 7
  • Works great also for Ubuntu 20.04. Thanks! – Yuan HOng Dec 19 '20 at 08:07
  • Your answer is also the only working way for me. – Daniel Aug 21 '21 at 17:34
  • 1
    [as dockershim deprecated in k8s](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.20.md#dockershim-deprecation) , I hosted a k8s cluster using containerd only, so this workaround won't work for my environment. – ws_ Sep 29 '21 at 03:35
  • 1
    for containerd users : 1. `sudo systemctl edit containerd` 2. add this text `[Service]LimitMEMLOCK=infinity` 3. `sudo systemctl daemon-reload && sudo systemctl restart containerd` – ws_ Sep 29 '21 at 06:58
  • this seems to have been done on specific nodes but what if we want to do this for all the nodes even when they are autoscaled using karpenter? – Chayan Bansal Jul 04 '23 at 07:59
0

If you use the Kubernetes you never need memlock!!!!

If you use ElasticSearch in Kubernetes, Then configure it with the following environment variable:

bootstrap.memory_lock=false

FALSE!!!

You need NOT set memlock in Kubernetes because Kubernetes does NOT run with swap-file.

Some applications (for example ElasticSearch) do not work correctly if some RAM given to them by the operating system is flushed to disk into the swap file. Therefore, these applications require you to block memory from being flushed to disk.

If swap-file is disabled in the operating system, then these applications will never encounter this problem. This is exactly the situation with Kubernetes, because it requires disabling the swap-file while install.

If you're using Kubernetes, then you do NOT need to block the memory flush to disk, as this will never happen.

Евгений Колпаков
  • 96
  • 7