Got error of "Too many open file", what's the ulimit when running the program using ProcessBuilder()...start()?

Question

I need to run a program after a Linux EC2 machine is provisioned on AWS. The following code will get "Too many open file" error. my_program will open a lot of files, maybe around 5000.

string cmd = "my_program";
Process process = new ProcessBuilder()
                      .inheritIO()
                      .command(cmd)
                      .start();

However, running my_program in the console can finish without any error. What's the ulimit when running the program using ProcessBuilder()...start()?

ulimit -n output 65535 in bash terminal.

Please add output of `ulimit -n` to your question (no comment). — Cyrus, Aug 14 '21 at 04:34
The value of a `ulimit` doesn't depend on Java. It is something that is set outside of your Java program. And your Java program can't change it. So, the answer to your question is "We can't tell you, but you can find it out for yourself using the `ulimit` command". — Stephen C, Aug 14 '21 at 04:39
@chrylis-cautiouslyoptimistic- it runs a pyspark script. The Spark code `spark.read.oprtion(...).parquet(*files)` opens a lot of files. — ca9163d9, Aug 14 '21 at 17:49
I suggest that your real problem may be a file descriptor leak. — user207421, Aug 16 '21 at 08:16
@user207421, the python code is actually pyspark code. How to tell if it's a file descriptor leak? — ca9163d9, Aug 26 '21 at 15:11

score 0 · Answer 1 · answered Aug 14 '21 at 10:35

First find out the limits your app has when running:

ps -ef | grep <<YOUR-APP-NAME>>

then:

cat /proc/<<PID-of-your-APP>>/limits

Here the problem is that you app. starts under X or Y user and these users have a different ulimit setup.

Check:

cat /etc/security/limits

... I think and increase those values.

Just my 2 cents...

score 0 · Answer 2 · answered Aug 16 '21 at 06:30

You need to ensure that you close() the files after use. They will be closed by the garbage collector (I'm not completely sure about this, as it can differ on each implementation) but if you process many files without closing them, you can run out of file descriptors right before the garbage collector has any chance to execute.

Another thing you can do is to use a resource based try statement, that will ensure that every resource you declare in the parenthesis group is a Closeable resource that will be forcibly close()d on exit from the try.

Anyway, if you want to rise the value of the maximum number of open files per process, just look at your shell's man page (bash(1) most probably) and search in it about the ulimit command (there's no ulimit manual page as it is an internal command to the shell, the ulimit values are per process, so you cannot start a process to make your process change it's maximum limits)

Beware that linux distributions (well, the most common linux distributions) don't have normally a way to configure an enforced value for this in a per user way (there's a full set of options to do things like this in BSD systems, but linux's login(8) program has not implemented the /etc/login.conf feature to do this) and rising arbitrarily this value can be a security problem in your system if it runs as a multiuser system.

Got error of "Too many open file", what's the ulimit when running the program using ProcessBuilder()...start()?

2 Answers2