7

We have two Nginx servers with independent local caches for caching images. Periodically there is problem with unmanaged growth of cache's size. This size may be far larger than max_size in nginx.conf. As a result cache occupies all free disk space.

I want to understand what the reason of this issue might be.

Proxy cache config (nginx.conf):

proxy_cache_path /opt2/nginx-cache-images1 max_size=150g levels=2:2 keys_zone=images1:1024m inactive=24h;
proxy_temp_path /opt2/proxy_temp 1 2;

Size of /opt2/nginx-cache-images1 is ~200GB.

Config of site location:

location / {
   proxy_set_header Host $host;
   proxy_set_header X-Forwarded-For $remote_addr;
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Is-Referer-Search-Engine $is_referer_search_engine;
   proxy_hide_header Set-Cookie;
   proxy_hide_header Content-Disposition;
   proxy_pass http://ua-image-proxy;
   default_type image/jpeg;

   proxy_cache images1;
   proxy_cache_key ua$request_uri$is_referer_search_engine;
   proxy_cache_valid 200 24h;
   proxy_cache_valid 301 24h;
   proxy_cache_valid 404 1h;
}

Nginx version: 1.7.7 and 1.7.9. It was compiled on Ubuntu 14.04.

Esa Jokinen
  • 46,944
  • 3
  • 83
  • 129
Vladyslav Zakhozhai
  • 71
  • 4
  • What is output of `du -kcd1 /opt2/nginx-cache-images1` ? – masegaloeh Apr 23 '15 at 06:05
  • After restarting nginx service all things goes ok. But periodically this issue happens. Full output of du - [link](https://docs.google.com/document/d/1XFLLjVmrthgxts_BKndBs5y4-YVv8fhoXD7YR_7clVs/edit?usp=sharing). – Vladyslav Zakhozhai Apr 23 '15 at 13:10
  • Also when the issue will happen I'll post another result of du command. – Vladyslav Zakhozhai Apr 23 '15 at 13:12
  • Now I have the same problem. Cache size is bigger than max_size. Result of du command you can see on the following link: [link](https://docs.google.com/document/d/1INni3IA8JEsGqT5v_xXTcoUVILun995NPfADIoYxakY/edit?usp=sharinghttps://docs.google.com/document/d/1INni3IA8JEsGqT5v_xXTcoUVILun995NPfADIoYxakY/edit?usp=sharing). Is it useful? – Vladyslav Zakhozhai Apr 23 '15 at 18:09
  • 1
    We have this on one of our busiest caches. It's like the cache manager just stops deleting anything for a period of time. Sometimes it recovers after a while and starts deleting a lot of files, only to start rising again. Often these situations seem to coincide with a burst of traffic. Judging by reports elsewhere this has been a problem since at least nginx 1.6.1. – GreenReaper Dec 28 '15 at 10:19

1 Answers1

2

For me this was fixed in nginx 1.13.1:

Workaround: now cache manager ignores long locked cache entries when cleaning cache based on the "max_size" parameter.

njahnke
  • 290
  • 3
  • 16