Docker long-running command memory consumption grows over time

Question

I am trying to run a long-running command in docker and I am seeing increasing memory consumption during the command execution.

I am running a test suite composed of 741 tests and 3390 assertions with PHPUnit in the context of a Laravel application. Most of those tests are integration tests which use a database connexion, redis and s3 services in docker-compose. I am producing a code coverage report with this test run. Here is the exact command I run

docker-compose run phpunit --verbose --testdox --colors=never --stop-on-error --stop-on-failure --coverage-clover coverage.xml

I have tried many ways to see where the memory was leaked (is it Docker, PHP or PHPUnit?). Here are the memory reports I set up with their results:

1. PHPUnit by defaults reports his memory usage at the end of the run. -> It was always around 80-100MB which is fine.
2. I used the teardown function and printed the memory_get_peak_usage. -> It was very similar to the previous something along 80-100 and it increased very slightly to 120MB in the later test. Seems fine
3. I added passthru('ps -o pid,user,%mem,command ax | sort -b -k3 -r | head') in the teardown function. -> I could see my memory consumption rise at very fast rate. I figured it matched the same percentage as line #4, thus inferring it was close to 6.5G by the end of the run. The only process using RAM was phpunit.
4. From my host, I would run docker stats --format "table {{.Name}}\t{{.MemPerc}} - {{.MemUsage}}\t{{.BlockIO}}" -> Same thing, memory increasing to 6.5 GB of RAM by the end of the run. The only container having increasing memory usage was the one running phpunit. Other containers from docker-compose didn't move (database, redis, s3, etc).
5. From my host, I would run watch -n 1 free -m to report RAM usage on my whole computer -> Same thing, memory increasing to 6.5 GB of RAM by the end of the run. As soon as the command completed, the 6.5 GB of RAM was instantly released.

My problem is very similar to this one: Docker does not free memory after creating and deleting files with PHP. I have tried the solutions proposed. I ran echo 3 > /proc/sys/vm/drop_caches and sync && sysctl -w vm.drop_caches=3 in privileged mode. The memory usage was not different, but I could see that the BlockIO was now increasing at the same rate than the memory usage which it did not do without one of those commands.

UPDATE april 28, 2021: It ended up not being a problem with Docker but a problem with PHPUnit. When the tests are run with processIsolation=false PHPUnit seems to be leaking. My understanding is that processIsolation=true, a new process is launched for every test case. So memory is freed between each tests, but then it takes way more time to execute.

processIsolation=false: Fast but leaky memory

processIsolation=true: Slow but leaked memory is release in between tests

Answer 1

Docker processes are not the one who are using the memory, docker is the one who is reporting the use. The memory is used by the the processes in the container and the kernel modules. Based on the information in the question, I don't see that you have problem, rather you are confused by the way that the memory is managed by the kernel, so I will try to explain on a very high level what is going on.

Kernel Namespaces

The modern Kernel allows creation of partitions (namespaces) where the processes that are assigned to this partitions can work in semi-isolation (isolated from all other processes in the user space, but not from all kernel subsystems/modules). Docker is using this feature of the kernel to create containers.

The container is running in a so called kernel namespace. The kernel namespace has little to do with the docker's memory space in which the docker application is running. The docker application is responsible only for the orchestration of the namespaces (creating new namespaces, assigning containers to namespaces, setting the resource limits, provisioning network infrastructure, etc.), so Docker, after creating the kernel namespace is not involved if further memory management of it.

So when you start a container, Docker creates a new namespace for the container, prapares all of the resource limitation, prepares the network and starts the processes in that name space. I have deliberately oversimplified the creation of the container for this conversation, so if you need more information you can check this posts:

• Understanding the Docker Internals
• Build Containers From Scratch in Go (Part 1: Namespaces)
• Demystifying Containers - Part I: Kernel Space

Are the Buffers and Cache really a problem

The kernel will try to use as much as possible from the "unused memory" as a cache. This makes a good sense. The unused memory is sitting idle, so using it as a cache to speed-up the processes is a really good strategy. This helps to make a huge difference for the processes that are I/O bound (mainly when processes have lot's of IO read operations). The article Linux Page Cache Basics provides good text as entry point to learn more about the kernel cache.

The kernel will tend repurpose the "free" memory for caches and buffers. As mentioned, this is very good feature and helps all processes to run faster. Once the memory pressure starts growing in the namespace, meaning the processes running in the namespace need more memory, the kernel will start shrinking the cache by finding the least used regions and releasing them so they can be allocated to the processes. This is described in the kernel documentation.

The kernel will not rush to release the cache unless there is a memory pressure. So unless there is a need for the memory, once allocated to the cache, the kernel will keep it. This also is a very good strategy from kernel perspective since it avoids additional work for releasing the cache and potentially repopulating it in future.

So, unless we talk about a defect in the kernel or in a kernel module, in general, the memory being used by the buffers/cache is a good thing and not an issue.

By using the sync && echo 3 > /proc/sys/vm/drop_caches which is explained in the kernel documentation section Documentation for /proc/sys/vm/ the system is forced to drop as much as possible cache objects which will lead toward I/O performance issues until the cache is repopulated. This is useful for debugging and profiling, but not as a good practice in a normal production. Once cache is cleaned, the kernel will need to repopulate (since there are other processes who really need this cache), which leads to the additional IO.

Managing the memory

Docker provides mechanism to manage how the container will use the memory and the limits associated with the container. You can check more in docker's documentation Runtime options with Memory, CPUs, and GPUs

Unless there is a special use case which demands special handing of the memory, the features that Docker provides for resource management are more than sufficient to assure smooth operations of the containers.

Troubleshooting the memory

Sometimes we can deal with memory leak problems, either by the kernel / kernel modules or by the applications in the container.

First step in troubleshooting is to identify who is really using the memory and how it's growing. To identify the problematic code, we need to troubleshoot the memory allocation and to understand if there are regions that are constantly growing and are not released when the memory pressure is reached. Apart from the free and ps the better information about the memory lifecycle can be found on the sysfs.

The sysfs exposes internals of the kernel as a file system for reading and sending signals. The /proc/sys/vm is also part of the same mechanism known as procfs. The /sys/fs/cgroup/memory is the folder where the information and the statistics about memory allocation. Docker creates it's own cgroup called Docker, so the statistics of all docker containers will be found in /sys/fs/cgroup/memory/docker/. Further more, docker will create a cgroup for for each container under /sys/fs/cgroup/memory/docker/<<container id>>.

To find the kernel statistics about the memory usage of a container you can inspect the files in the sysfs as follows:

• /sys/fs/cgroup/memory/docker/<<container id>>/memory.stat statistics about the memory usage (similar to free but with more details
• /sys/fs/cgroup/memory/docker/<<container id>>/memory.limit_in_bytes current set limits in bytes for the container
• /sys/fs/cgroup/memory/docker/<<container id>>/memory.kmem.slabinfo regions of the kernel memory and the usage for the cache/buffer objects