1

I'm getting Too Many Open Files issue even when I'm closing all files and connections I'm opening and when I've set the soft and hard limit to max available i.e. 65535. So in order to debug, I'm trying to re-order the output of lsof in terms of time.

hckrman
  • 136
  • 9

2 Answers2

0

If you can control the launch command, consider using strace/ltrace. It can show you each file open/close.

# When using ltrace, specify stdlib functions to monitor
ltrace -e open+close YOUR_COMMAND_HERE
OR
# Modern linux system will use openat system call to open a file
strace -e openat,close YOUR_COMMAND_HERE

You can also attach strace (and ltrace) to a running process (see man page)

If you know that you have unclosed (network) connections (and not files), consider extending/replacing the filter to focus on accept (inbound connection), connect (outbound connection) as needed.

dash-o
  • 13,723
  • 1
  • 10
  • 37
0

If the strace/ltrace is overkill, consider inspecting the /proc/PID/fd folder. Each entry modification time will show the timestamp that the FD was created, effectively the open/connect/accept time:

# X_PID is the PID of the process to monitor/check
ls -lt --time-style=full-iso /proc/$X_PID/fd

Sample Output

total 0
lrwx------ 1 owner owner 64 2019-10-16 16:25:14.359506339 +0300 0 -> /dev/pts/0
lrwx------ 1 owner owner 64 2019-10-16 16:25:14.359506339 +0300 1 -> /dev/pts/0
l-wx------ 1 owner owner 64 2019-10-16 16:25:32.183370423 +0300 11 -> /tmp/a
l-wx------ 1 owner owner 64 2019-10-16 16:26:47.862798157 +0300 15 -> /tmp/b
l-wx------ 1 owner owner 64 2019-10-16 16:26:47.862798157 +0300 16 -> /tmp/a
l-wx------ 1 owner owner 64 2019-10-16 16:27:20.918550476 +0300 17 -> /tmp/a
lrwx------ 1 owner owner 64 2019-10-16 16:25:14.359506339 +0300 2 -> /dev/pts/0
lrwx------ 1 owner owner 64 2019-10-16 16:25:14.359506339 +0300 255 -> /dev/pts/0
dash-o
  • 13,723
  • 1
  • 10
  • 37