Nginx content caching causing Docker memory spike

Question

I'm trying to set up proxy content caching with Nginx inside of Docker, but am experiencing memory issues with my container. The actual Nginx implementation works fine (pages are being cached and served as expected), but as soon as pages start being cached, my container memory (measured with "docker stats") climbs extremely quickly.

I would expect about a 1MB increase for every 8,000 pages cached as per the Nginx docs (https://www.nginx.com/blog/nginx-caching-guide/), but the growth is far greater - probably around 40MB every 8000 pages. Additionally, when running "top" inside my container, the nginx process memory looks normal - a couple MB - while my container memory is skyrocketing.

It almost seemed liked the cached pages themselves, which are stored in a specific directory, are taking up memory? This shouldn't be the case, as only the cache keys should be in memory. I think I've tested to around 25,000 pages being cached - container memory never falls off. Additionally, if I'm just proxying requests with caching turned off, there is no container memory spike.

I'm running an extremely basic nginx configuration setup - pretty much what is detailed in the Nginx docs link.

proxy_cache_path /path/to/cache levels=1:2 keys_zone=my_cache:10m max_size=10g
             inactive=60m use_temp_path=off;
server {
    ...
    location / {
        proxy_cache my_cache;
        proxy_pass http://my_upstream;
    }
}

Docker images tested - official nginx image, alpine:3.4 with nginx installed, centos:7 with nginx installed

Docker versions tested: Docker for Mac 1.12.1, Docker 1.11.2 (on Kubernetes)

Grafana dashboard showing memory growth

Could this possibly be the filesystem cache? What does `free` say, when you run it from within the container (pay attention to the `-/+ buffers/cache` line)? — helmbert, Dec 08 '16 at 22:22
When I run `free`, on the `-/+ buffers/cache` line, under the "free" column", the memory available decreases extremely slowly - about what I would expect. On the `Mem:` line, under the "cached" column, the value is increasing rapidly. So that means the OS is caching each entire page in memory, but if an application process needs memory, the OS page cache will be emptied as needed? — martialartsandcrafts, Dec 08 '16 at 23:52

score 4 · Accepted Answer · edited May 23 '17 at 12:09

Wow, I did pretty much the same (checking docker stats and then using graphana with cadvisor and influxDB to plot the increase) with my application(not nginx). And I agree with your conclusion that page cache is contributing to that increase in memory. After some digging into cgroups metrics for that container, I solved my own question: https://stackoverflow.com/a/41687155/6919159

If you set a limit to the container's memory usage as described in my answer, you should see the container reclaiming memory. Hope it helps, though its been 2 months!

Nginx content caching causing Docker memory spike

1 Answers1