0

I'm running a Laravel queue job called "webhooks" using beanstalkd and supervisord on my Ubuntu server. I can see the job properly running with process ID 4403:

webhooks                         RUNNING    pid 4403, uptime 4 days, 19:47:01

As you can see, this job has been running for 4 days. In my error logs, I started to notice the following error appearing:

error:02001018:system library:fopen:Too many open files

When I ran lsof | php to see what files were open, I noticed a ton of files open that had type FIFO. Here's an expert from the output:

COMMAND     PID   TID       USER   FD      TYPE             DEVICE SIZE/OFF       NODE NAME     
php        4403             root    0r     FIFO                0,8      0t0    9215811 pipe
php        4403             root    1w     FIFO                0,8      0t0    9215812 pipe
php        4403             root    2w     FIFO                0,8      0t0    9215812 pipe
php        4403             root    3w     FIFO                0,8      0t0    9215812 pipe
php        4403             root    4w     FIFO                0,8      0t0    9215812 pipe
php        4403             root    8r     FIFO                0,8      0t0    9215811 pipe
php        4403             root    9r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   10r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   11r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   12r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   13r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   14r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   15r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   16r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   17r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   18r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   19r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   20r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   21r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   22r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   23r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   24r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   25r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   26r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   27r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   28r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   29r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   30r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   31r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   32r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   33r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   34r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   35r     FIFO                0,8      0t0    9215811 pipe
php        4403             root   36r     FIFO                0,8      0t0    9215811 pipe

This is just an excerpt. There are actually around 1200 of these FIFO files open.

Does anyone know what is causing the files to be created and why they are remaining open?

flyingL123
  • 245
  • 4
  • 12
  • That number of open files is normal, under 10k everything is normal. You need to increase the number of open files per user and process to function at better parameters. And you should check if you close your connections properly, as those are for pipes, and that is a connection in your PHP. – Pentium10 Jun 02 '15 at 09:01
  • Thank you. Won't I still have the same problem if I increase the allowed number of open files? It will just take longer to get there, but I will eventually hit the limit. It seems that the number of open pipes just keeps growing and they are never actually closed. I'm not actually opening or closing these files anywhere. I think it's all being done behidn the scenes by beanstalkd and supervisord. Is it possible there is a bug in those libraries causing this problem? – flyingL123 Jun 04 '15 at 13:07
  • If it's keep growing you have an issue somewhere in your code. I never see this growing stuff if the protocol is implemented correctly. You have to close some thing for sure. – Pentium10 Jun 04 '15 at 13:44
  • Do you have any experience with beastalkd or supervisord? Do you know what part of these libraries would be opening these pipes and not closing them? – flyingL123 Jun 04 '15 at 13:47
  • I do have a lot of experience with beanstalkd, we use pheanstalk library to connect, we didn't faced the issue you mentioned. I don't have experience with supervisord. – Pentium10 Jun 04 '15 at 13:51

0 Answers0