2

I have configured a 2 node cluster with hadoop and installed hbase. It was working properly and I have run some basic map reduce jobs in hadoop and I was able to create and list some tables in hbase too. However I have few data in hdfs/hbase and there were no job running. After a while I started to get "Java.net.Socket: Too many open files" error in hbase logs.

I have looked for some solutions but there are mainly answers about increasing limit. However I am curious about why there are too many open files. This cluster is not used by any other program and I have not run any job other than simple map reduce tasks in tutorials.

Why could it be?

EDIT

After Andrzej suggested, I have run this command (lsof | grep java) and I have observed that there are lots of connection in different ports which are waiting to be closed. This is just a few line of the output of the command

java      29872     hadoop  151u     IPv6          158476883      0t0       TCP os231.myIP:44712->os231.myIP:50010 (CLOSE_WAIT)
java      29872     hadoop  152u     IPv6          158476885      0t0       TCP os231.myIP:35214->os233.myIP:50010 (CLOSE_WAIT)
java      29872     hadoop  153u     IPv6          158476886      0t0       TCP os231.myIP:39899->os232.myIP:50010 (CLOSE_WAIT)
java      29872     hadoop  155u     IPv6          158476892      0t0       TCP os231.myIP:44717->os231.myIP:50010 (CLOSE_WAIT)
java      29872     hadoop  156u     IPv6          158476895      0t0       TCP os231.myIP:44718->os231.myIP:50010 (CLOSE_WAIT)

Now the question becomes, why do not they close automatically if the connection is useless now? If they do not get aborted automatically, is there any way to close them with a crontab script or something similar?

Thanks

Sami
  • 490
  • 6
  • 29
  • You have limit in kernel, limit in SELinux and limit open files for service. See https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/ or https://naveensnayak.wordpress.com/2015/09/17/increasing-file-descriptors-and-open-files-limit-centos-7/ It depends on your OS. – Andrzej Jozwik Jan 02 '17 at 07:44
  • I know the limit. But I am wondering why there have been that much open file. increasing the limit can be a temporary solution. Is there some open files which should have been closed etc? – Sami Jan 02 '17 at 08:00
  • 2
    If you use linux call `lsof | grep java` . – Andrzej Jozwik Jan 02 '17 at 08:33
  • possibly this could be pointer to investigate further. Sami : pls paste the result of above command by @andrzej-jozwik – Ram Ghadiyaram Jan 02 '17 at 09:41
  • what is the version of hbase you are using? Isnt it [HBASE-9393](https://issues.apache.org/jira/browse/HBASE-9393) – Ram Ghadiyaram Jan 02 '17 at 13:52
  • It is hbase 0.94.1, bt I got the same problem in 1.2.4 too – Sami Jan 03 '17 at 07:03

1 Answers1

0

... I am curious about why there are too many open files?...

HBase keeps open all the files all the time. Here is some example. If you have 10 tables with 3 column familes each with average of 3 files per column family and 100 regions per Region Server per table, there will be 10*3*3*100 = 9000 file descriptors open. This math doesn't take in account JAR files, temp files etc.

Suggested value for ulimit is 10240, but you might want to set it to a value that matches better your case.

Ivan Georgiev
  • 1,115
  • 7
  • 11