2

I have a script using docker python library or Docker Client API. I would like to limit each docker container to use only 10cpus (total 30cpus in the instance), but I couldn't find the solution to achieve that.

I know in docker, there is --cpus flag, but docker only has cpu_shares (int): CPU shares (relative weight) parameter to use. Does everyone have experience in setting the limit on cpu usage using docker?

import docker
client = docker.DockerClient(base_url='unix://var/run/docker.sock')
container = client.containers.run(my_docker_image, mem_limit=30G)

Edits:

I tried nano_cpus as what here suggests, like client.containers.run(my_docker_image, nano_cpus=10000000000) to set 10CPUS. When I inspectED the container, it did show "NanoCpus": 10000000000". However, if I run the R in the container and do parallel::detectCores(), it still shows 30, which I am confused. I also link R tag now.

Thank you!

WenliL
  • 419
  • 2
  • 14
  • 1
    Setting a CPU limit will keep the kernel from allocating your container more than that amount of processing time, but it won't make it look like there are fewer cores. Your setup is probably correct and if you run it under load it shouldn't actually use more than 10 CPU worth. – David Maze Aug 26 '22 at 14:22
  • @DavidMaze Thanks. I also noticed that just now, setting the `nano_cpus ` is managed by `cgroups`. If I use `parallelly` package in r, and run `parallelly::detectCores()`, it can give me the correct cpu values. – WenliL Aug 26 '22 at 14:36

2 Answers2

0

All that I see about this is with operating systems. Is there a reason you can't just use these on system? Just asking for clarification here.

If you can use the cmd, then you can use this command:

sudo docker run -it --cpus=1.0 alpine:latest /bin/sh
Joe Musashi
  • 28
  • 7
  • Thanks. The reason is we already have many infra setup using the `docker-py`, so we prefer not to make huge big changes to just replace with CLI. – WenliL Aug 26 '22 at 03:16
  • btw, I also tried to check cpus in your example container ( `docker run -it --cpus=1.0 alpine:latest /bin/sh`). It doesn't match the `--cpus` value. / # nproc --all 4 – WenliL Aug 26 '22 at 03:27
  • try this article: https://medium.com/codex/how-to-limit-cpu-and-memory-usage-in-docker-containers-a024b429f55e?adlt=strict&toWww=1&redig=A53D6E0F8473461AA587A7671B5386B9 – Joe Musashi Aug 26 '22 at 03:30
0

Setting nana_cpus works and you could use parallelly::availableCores() to detect cpus set by cgroup in r.

WenliL
  • 419
  • 2
  • 14