5
tail: inotify cannot be used, reverting to polling: Too many open files

I'm running apache and tomcat servers on Ubuntu (AWS ec2). Whenever I try to tail the catalina.out of tomcat I get too many open files. However I am able to view it using vi.

After searching around on internet, I tried the following command:

lsof | awk '{ print $2; }' | sort -rn | uniq -c | sort -rn | head

With results below 

 17 5650
 17 5178
 13 5972
 10 5976
 10 5974
  9 5977
  9 5975
  9 5973
  8 5978
  4 9

When I just ran lsof the process ids: 5650 were bash, 5178 was again bash and others were sshd,top and apache2.

Why is there bash, top, sshd opening files in huge number? How can I close these files? Will killing these processes do any good ? Will the number decrease by itself or do I have to do anything? Right now everything is working as expected except that tail -f gives me too many open files.

I use top and ssh to server a lot. But why don't they release the files? OR am I connecting the wrong dots.

user23577
  • 71
  • 1
  • 1
  • 6
  • do you have any cronjobs running (try crontab -l)? also try ps aux | grep -E '5650|5178' to see if there is more info about those bash processes. – suitablyawesome May 24 '13 at 21:04
  • tried crontab -l no crontab for ubuntu. for ps aux | grep -E '5650|5178' ubuntu 5650 0.0 1.2 25132 7696 pts/0 Ss 20:24 0:01 -bash – user23577 May 24 '13 at 21:31
  • @Toquonce I also did a lsof |grep tail | wc -l 20 So my understanding is that there are 20 files open by tail which is still not the number of open files limit described in ulimit -a (1024) – user23577 May 24 '13 at 21:43

3 Answers3

7

Probably you ran out of inotify watches. By default it's an absurdly low 8192.

Check your current value by:

sysctl fs.inotify.max_user_watches

Then change it to something more reasonable by editing /etc/sysctl.conf or a file which it includes, and adding:

fs.inotify.max_user_watches = 524288

(or whatever value) and then running sysctl -p to have it take effect.

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
  • Thanks for your reply. indeed the number is 8192. But I'm wondering would the inotify watches would free themselves if no process is using them. Since no tail is currently running, wouldn't the inotify watches be available. And is there any side effect of increasing the max user watches. – user23577 May 25 '13 at 00:23
  • I also ran sudo sysctl fs.inotify.max_user_watches=524288 sysctl -p then ran sysctl fs.inotify.max_user_watches , the number was 524288. However when i still run tail i get the same inotify error – user23577 May 25 '13 at 00:33
  • 3
    However if i change max_user_instances from 128 to 256 , it works. Wondering if there is a side effect of it. And what causes it to reach the limit. – user23577 May 25 '13 at 00:42
2

You can try increasing fs.inotify.max_user_instances:

sysctl fs.inotify.max_user_instances=512
user1948518
  • 21
  • 1
0

Maybe it's an Ubuntu kernel bug, check this from launchpad bugtracker.

And if needed upgrade your kernel !

Zskdan
  • 121
  • 1