What are the number of open files for a user on linux and system wide for linux?

Question

Sorry, this question has several layers but all deal with the number of open files.

I'm getting a "Too many open files" message in my application log for the app we're developing. Someone suggested to me to:

1. Find the number of open files currently being used, system wide and per user
2. Find what the limit for open files of the system and user are.

I ran ulimit -n and it returned 1024. I also looked at /etc/limits.conf and there isn't anything special in that file. /etc/sysctl.conf is also not modified. I'll list the contents of the files below. I also ran lsof | wc -l, which returned 5000+ lines (if I'm using it correctly).

So, my main questions are:

1. How do I find the number of open files allowed per user? Is the soft limit the nofile setting found/defined in /etc/limits.conf? What is the default since I didn't touch /etc/limits.conf?
2. How do I find the number of open files allowed system-wide? Is it the hard limit in limits.conf? What's the default number if limits.conf isn't modified?
3. What is the number that ulimit returns for open files? It says 1024 but when I run lsof and count the lines, it's over 5000+ so something is not clicking with me. Are there other cmds I should run or files to look at to get these limits? Thanks in advance for your help.

Content of limits.conf

# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#
#<domain>        <type>  <item>  <value>
#
#Where:
#<domain> can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
#
#<type> can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
#
#<item> can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open files
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit (KB)
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#        - nice - max nice priority allowed to raise to values: [-20, 19]
#        - rtprio - max realtime priority
#
#<domain>      <type>  <item>         <value>
#

#*               soft    core            0
#*               hard    rss             10000
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
#@student        -       maxlogins       4

# End of file

Content of sysctl.conf

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

# the interval between the last data packet sent and the first keepalive probe
net.ipv4.tcp_keepalive_time = 600

# the interval between subsequential keepalive probes
net.ipv4.tcp_keepalive_intvl = 60

# the interval between the last data packet sent and the first keepalive probe
net.ipv4.tcp_keepalive_time = 600

# the interval between subsequential keepalive probes
net.ipv4.tcp_keepalive_intvl = 60

# the number of unacknowledged probes to send before considering the connection dead and notifying the application layer
net.ipv4.tcp_keepalive_probes = 10

# the number of unacknowledged probes to send before considering the connection dead and notifying the application layer
net.ipv4.tcp_keepalive_probes = 10

# try as hard as possible not to swap, as safely as possible
vm.swappiness = 1
fs.aio-max-nr = 1048576
#fs.file-max = 4096

Answer 1

There is no per-user file limit. The ones you need to be aware of are system-wide and per-process. The files-per-process limit multiplied by the processes-per-user limit could theoretically provide a files-per-user limit, but with normal values the product would be so large as to be effectively unlimited.

Also, the original purpose of lsof was to LiSt Open Files, but it has grown and lists other things now, like cwd and mmap regions, which is another reason for it to output more lines than you expect.

The error message "Too many open files" is associated with errno value EMFILE, the per-process limit, which in your case appears to be 1024. If you can find the right options to limit lsof to just displaying actual file descriptors of a single process, you'll probably find that there are 1024 of them, or something very close.

The system-wide file descriptor limit rarely needs to be manually adjusted these days, since its default value is proportional to memory. If you need to, you can find it at /proc/sys/fs/file-max and information about current usage at /proc/sys/fs/file-nr. Your sysctl file has a value of 4096 for file-max, but it's commented out so you shouldn't take it seriously.

If you ever manage to hit the system-wide limit, you'll get errno ENFILE, which translates to the error message "File table overflow" or "Too many open files in system".