11

First time setting up a VPS – being mindful not to ask here unless I done due diligence and provide context.

On my remote VPS, through the terminal, almost all commands I run end up with an Error: Too many open files message and I need your help to move forward.

I run: CentOS Linux release 7.6.1810 (Core) on a machine with 1 CPU core and 2048Mb RAM. It has been setup with a LEMP stack Nginx 1.16.1, PHP-FPM 7.3.9, MariaDb 10.4.8 intended for a simple wordpress site.

I have tried:

  1. Google and forum searches.
  2. Applied these settings (manually restarting VPS manually via control panel each time):

System-wide settings in /etc/security/limits.conf:

nginx       soft    nofile      1024
nginx       hard    nofile      65536
root        hard    nofile      65536
root        soft    nofile      1024

adjustments to memory limits and uploads in /etc/php.ini:

memory_limit = 256M
file_uploads = On
upload_max_filesize = 128M
max_execution_time = 600
max_input_time = 600
max_input_vars = 3000

PHP rlimit settings in /etc/php-fpm.d/www.conf:

rlimit_files = 65535

Setting NGINX limits (and other settings) in nginx.conf:

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  10000;
}

worker_rlimit_nofile 100000;


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;
    client_body_buffer_size 128k;
    client_header_buffer_size 10k;
    client_max_body_size 100m;
    large_client_header_buffers 4 256k;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*.conf;
    server_names_hash_bucket_size 64;
}

Here is the output of cat /proc/sys/fs/file-nr:

45216   0   6520154

Here is the output of ps aux|grep nginx|grep -v grep:

root       928  0.0  0.0  46440  1192 ?        Ss   00:25   0:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nginx      929  0.0  0.2  50880  6028 ?        S    00:25   0:00 nginx: worker process
nginx     9973  0.0  0.1 171576  4048 ?        S    04:28   0:00 php-fpm: pool www
nginx     9974  0.0  0.1 171576  4048 ?        S    04:28   0:00 php-fpm: pool www
nginx     9975  0.0  0.1 171576  4048 ?        S    04:28   0:00 php-fpm: pool www
nginx     9976  0.0  0.1 171576  4048 ?        S    04:28   0:00 php-fpm: pool www
nginx     9977  0.0  0.1 171576  4052 ?        S    04:28   0:00 php-fpm: pool www

Switching user to nginx with su - nginx and checking limits with: ulimit -Sn returns 1024 ulimit -Hnreturns 65536

The lsof | wc -l command returns: 4776

Hope you can help steer me in the right direction to solve the Too Many Files problem!

EDIT - the following command shows more info:

service nginx restart

Redirecting to /bin/systemctl restart nginx.service
Error: Too many open files
Job for nginx.service failed because a configured resource limit was exceeded. See "systemctl status nginx.service" and "journalctl -xe" for details.
[root@pars ~]# systemctl status nginx.service
● nginx.service - nginx - high performance web server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/nginx.service.d
           └─worker_files_limit.conf
   Active: failed (Result: resources) since Fri 2019-09-13 05:32:23 CEST; 14s ago
     Docs: http://nginx.org/en/docs/
  Process: 1113 ExecStop=/bin/kill -s TERM $MAINPID (code=exited, status=0/SUCCESS)
  Process: 1125 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
 Main PID: 870 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/virtualizor.service/system.slice/nginx.service

Sep 13 05:32:22 pars.work systemd[1]: Starting nginx - high performance web server...
Sep 13 05:32:22 pars.work systemd[1]: PID file /var/run/nginx.pid not readable (yet?) after start.
Sep 13 05:32:22 pars.work systemd[1]: Failed to set a watch for nginx.service's PID file /var/run/nginx.pid: Too many open files
Sep 13 05:32:23 pars.work systemd[1]: Failed to kill control group: Input/output error
Sep 13 05:32:23 pars.work systemd[1]: Failed to kill control group: Input/output error
Sep 13 05:32:23 pars.work systemd[1]: Failed to start nginx - high performance web server.
Sep 13 05:32:23 pars.work systemd[1]: Unit nginx.service entered failed state.
Sep 13 05:32:23 pars.work systemd[1]: nginx.service failed.
Henrik Söderlund
  • 231
  • 1
  • 2
  • 8

4 Answers4

13

It is not actually open file handles that have run out, but inotify watches.

You can see this in the error message:

Sep 13 05:32:22 pars.work systemd[1]: Failed to set a watch for nginx.service's PID file /var/run/nginx.pid: Too many open files

To solve the problem, you need to raise the number of inotify watches the system has available. If you actually check, you will find it has some ridiculously low value like 8192.

$ sysctl fs.inotify.max_user_watches
fs.inotify.max_user_watches = 8192

You can set the sysctl fs.inotify.max_user_watches to a higher value persistently by editing /etc/sysctl.conf or creating a file in the /etc/sysctl.d directory. For example, my system has:

$ cat /etc/sysctl.d/10-user-watches.conf 
fs.inotify.max_user_watches = 1048576

And then load it with sysctl -p.

You may not want to go straight to that number and cause the kernel to allocate memory to track a million user file watch slots; instead, just take the current value and double it until the problem stops occurring.

bmike
  • 283
  • 1
  • 19
Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
  • sounds great, tried adding it to /etc/sysctl.conf and loading it with sysctl -p without success, checking with your command upon reboot shows the same 8192 value. Any hints? What do I name the new file you suggest? – Henrik Söderlund Sep 13 '19 at 08:21
  • 2
    @HenrikSöderlund You could use the path I suggested in the answer. – Michael Hampton Sep 13 '19 at 08:46
  • I followed your instruction, created a similar file and hit the sysctl -p command after as well, but the value is still stuck at 8192 I'm afraid. I did add in the fs.file-max=100000 in the file as well on a separate line btw. Any ideas what might be preventing it from loading the setting? – Henrik Söderlund Sep 13 '19 at 12:24
  • extra info: It seems to state "Read-only file system" whenever I run sysctl-p or try other copy/paste commands from google with echo echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p etc. – Henrik Söderlund Sep 13 '19 at 12:34
  • What happens if you set the value first with ‘sysctl fs.inotify.max_user_watches=1048576’? – tolien Sep 13 '19 at 15:50
  • 1
    @HenrikSöderlund It sounds like you are not on a real virtual machine, but on an OpenVZ container. In this case you might not be able to change the sysctl. You can confirm the virtualization type by running the command `hostnamectl` and viewing the output. If you do have OpenVZ you probably will need to go shopping for another VPS (and avoid OpenVZ). – Michael Hampton Sep 13 '19 at 17:45
  • @tolien this is printed on screen then: sysctl: setting key "fs.inotify.max_user_watches": Read-only file system – Henrik Söderlund Sep 14 '19 at 00:10
  • @MichaelHampton the command did not work, however, on the specs i can see "LXC Virtualization" - does this help? – Henrik Söderlund Sep 14 '19 at 00:12
  • @MichaelHampton virt-what gives: lxc – Henrik Söderlund Sep 14 '19 at 03:49
  • @HenrikSöderlund Yes, that's another container. You won't be able to solve this problem here. Find another VPS. – Michael Hampton Sep 14 '19 at 07:15
  • thanks @MichaelHampton, good to finally get to the root of the problem, bummer with how it needs to be solved! ;-) I will mark your helpful answer/comments as the solution in this case, many thanks! – Henrik Söderlund Sep 15 '19 at 06:04
  • I am on a real VM and the suggested solutions don't change anything for me either, but also without error message... – xeruf Nov 08 '22 at 09:27
2

To change ulimit settings for services you need to modify the systemd unit. 

sudo systemctl edit --full nginx.service

And add the desired value to the service section

[Service]
LimitNOFILE=<integer>
...
HBruijn
  • 77,029
  • 24
  • 135
  • 201
  • Thanks for your reply, i did follow your suggestion and edited the file following your command, I set: LimitNOFILE=65536 - problem still persists I am afraid. Meanwhile I am following Michael's reply above which is not yet fully there – Henrik Söderlund Sep 13 '19 at 12:27
0

Same issue was with Debian 9 as well as Debian 10 buster.

The error was shown:

Failed to allocate directory watch: Too many open files

and increasing number of open files in Linux, didn't help, it was already maxed out:

fs.file-max = 9223372036854775807

The fix is to increase user instances count from 128 till something like this or more:

sysctl fs.inotify.max_user_instances=1024

and making it permanent as well with watches:

sudo bash -c 'cat <<EOF> /etc/sysctl.d/fs_inotify.conf
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576
EOF'
Arunas Bartisius
  • 709
  • 1
  • 7
  • 13
0

Create a new file at /etc/security/limits.d/NN-custom.conf (replace NN with some number such as 20,30 ..etc). Name of the file does not matter but it should have extension .conf

Put the following contents in the above file (these are same as you mentioned in your post ,adjust as per your needs):

 nginx       soft    nofile      1024 
 nginx       hard    nofile      65536 
 root        hard    nofile      65536 
 root        soft    nofile      1024

run# sysctl -p /etc/security/limits.d/NN-custom.conf

Log off and log back in and restart your service.

Hope this helps.

skb007
  • 136
  • 4
  • 6