Too much open files Exception under "unlimited" system

Question

I am seing a lot of too many open files exceptions in the execution of my program. Typically those occur in the following form:

org.jboss.netty.channel.ChannelException: Failed to create a selector.

...
Caused by: java.io.IOException: Too many open files

However, those are not the only exceptions. I have observed similar ones (caused by "too many open files") but those are much less frequent.

Strangely enough i have set the limit of open files of the screen session (from where i launch my programs) as 1M:

root@s11:~/fabiim-cbench# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
**open files                      (-n) 1000000**
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Moreover, as observed by the output of lsof -p I see no more that 1111 open files (sockets, pipes, files) before the exceptions are thrown.

Question: What is wrong and/or how can i dig deeper into this problem.

Extra: I am currently integrating Floodlight with bft-smart. In a nutshell the floodlight process is the one crashing with too much open files exceptions when executing a stress test launched by a benchmark program. This benchmark program will maintain 64 tcp connections to the floodlight process which in turn should maintain at least 64 * 3 tcp connections to the bft-smart replicas. Both programs use netty to manage these connections.

I don't known if I am running jboss (I think netty and jboss are two separate things). But I am running every process as root. — fabiim, Mar 12 '14 at 15:05

andrewdotn · Accepted Answer · 2014-03-12T16:23:09.647

First thing to check—can you run ulimit from inside your Java process to make sure that the file limit is the same inside? Code like this should work:

InputStream is = Runtime.getRuntime().exec(new String[] {"bash", "-c", "ulimit -a"}).getInputStream();
int c;
while ((c = is.read()) != -1) {
    System.out.write(c);
}

If the limit still shows 1 million, well, you’re up for some hard debugging.

Here are a couple of things that I would look into if I had to debug this—

Are you running out of tcp port numbers? What does netstat -an show when you hit this error?
Use strace to find out exactly what system call with what parameters is causing this error to be thrown. EMFILE is a return value of 24.

The “Too many open files” EMFILE error can actually be thrown by a number of different system calls for a number of different reasons:

$ cd /usr/share/man/man2
$ zgrep -A 2 EMFILE *
accept.2.gz:.B EMFILE
accept.2.gz:The per-process limit of open file descriptors has been reached.
accept.2.gz:.TP
accept.2.gz:--
accept.2.gz:.\" EAGAIN, EBADF, ECONNABORTED, EINTR, EINVAL, EMFILE,
accept.2.gz:.\" ENFILE, ENOBUFS, ENOMEM, ENOTSOCK, EOPNOTSUPP, EPROTO, EWOULDBLOCK.
accept.2.gz:.\" In addition, SUSv2 documents EFAULT and ENOSR.
dup.2.gz:.B EMFILE
dup.2.gz:The process already has the maximum number of file
dup.2.gz:descriptors open and tried to open a new one.
epoll_create.2.gz:.B EMFILE
epoll_create.2.gz:The per-user limit on the number of epoll instances imposed by
epoll_create.2.gz:.I /proc/sys/fs/epoll/max_user_instances
eventfd.2.gz:.B EMFILE
eventfd.2.gz:The per-process limit on open file descriptors has been reached.
eventfd.2.gz:.TP
execve.2.gz:.B EMFILE
execve.2.gz:The process has the maximum number of files open.
execve.2.gz:.TP
execve.2.gz:--
execve.2.gz:.\" document ETXTBSY, EPERM, EFAULT, ELOOP, EIO, ENFILE, EMFILE, EINVAL,
execve.2.gz:.\" EISDIR or ELIBBAD error conditions.
execve.2.gz:.SH NOTES
fcntl.2.gz:.B EMFILE
fcntl.2.gz:For
fcntl.2.gz:.BR F_DUPFD ,
getrlimit.2.gz:.BR EMFILE .
getrlimit.2.gz:(Historically, this limit was named
getrlimit.2.gz:.B RLIMIT_OFILE
inotify_init.2.gz:.B EMFILE
inotify_init.2.gz:The user limit on the total number of inotify instances has been reached.
inotify_init.2.gz:.TP
mmap.2.gz:.\" SUSv2 documents additional error codes EMFILE and EOVERFLOW.
mmap.2.gz:.SH AVAILABILITY
mmap.2.gz:On POSIX systems on which
mount.2.gz:.B EMFILE
mount.2.gz:(In case no block device is required:)
mount.2.gz:Table of dummy devices is full.
open.2.gz:.B EMFILE
open.2.gz:The process already has the maximum number of files open.
open.2.gz:.TP
pipe.2.gz:.B EMFILE
pipe.2.gz:Too many file descriptors are in use by the process.
pipe.2.gz:.TP
shmop.2.gz:.\" SVr4 documents an additional error condition EMFILE.
shmop.2.gz:
shmop.2.gz:In SVID 3 (or perhaps earlier)
signalfd.2.gz:.B EMFILE
signalfd.2.gz:The per-process limit of open file descriptors has been reached.
signalfd.2.gz:.TP
socket.2.gz:.B EMFILE
socket.2.gz:Process file table overflow.
socket.2.gz:.TP
socketpair.2.gz:.B EMFILE
socketpair.2.gz:Too many descriptors are in use by this process.
socketpair.2.gz:.TP
spu_create.2.gz:.B EMFILE
spu_create.2.gz:The process has reached its maximum open files limit.
spu_create.2.gz:.TP
timerfd_create.2.gz:.B EMFILE
timerfd_create.2.gz:The per-process limit of open file descriptors has been reached.
timerfd_create.2.gz:.TP
truncate.2.gz:.\" error conditions EMFILE, EMULTIHP, ENFILE, ENOLINK.  SVr4 documents for
truncate.2.gz:.\" .BR ftruncate ()
truncate.2.gz:.\" an additional EAGAIN error condition.

If you check out all these manpages by hand, you may find something interesting. For example, I think it’s interesting that epoll_create, the underlying system call that is used by NIO channels, will return EMFILE “Too many open files” if

The per-user limit on the number of epoll instances imposed by /proc/sys/fs/epoll/max_user_instances was encountered. See epoll(7) for further details.

Now that filename doesn’t actually exist on my system, but there are some limits defined in files in /proc/sys/fs/epoll and /proc/sys/fs/inotify that you might be hitting, especially if you’re running multiple instances of the same test on the same machine. Figuring out if that’s the case is a chore in itself—you could start by checking syslog for any messages…

Good luck!

Too much open files Exception under "unlimited" system

1 Answers1