2

I have a .jar file that I want to run on a supercomputer. There are some 40 CPU nodes available but Java uses only one of them when running my program. Is there any way to tell Java to use all the available nodes to run a given program (preferably without recompiling the program)?

Reza
  • 388
  • 2
  • 14
  • Update to Java8, which has functionalities that make it possible to use several nodes – Stultuske Sep 22 '15 at 13:36
  • The free lunch is over http://www.gotw.ca/publications/concurrency-ddj.htm – Sleiman Jneidi Sep 22 '15 at 13:37
  • 3
    Also, your software is multi-threaded, right? Upgrading to Java 8 won't give you anything if your software runs in one thread anyway. What you would do then is start multiple instances, one per core, and ensure that they are properly orchestrated, if required. – cjstehno Sep 22 '15 at 13:42
  • Yes, it is. I can specify the number of threads the program uses (which I set to a pretty large number). But it still uses only one core (JRE's fault I guess). – Reza Sep 22 '15 at 13:48
  • How are you distributing the work over the available threads? – John McClean Sep 22 '15 at 14:17

1 Answers1

2

Java always uses all the available CPUs by default. By default when you start it creates a thread for every logical CPU just for performing the GC.

There are some 40 CPU nodes available but Java uses only one of them when running my program

The problem is that your program only uses one thred because that is how you wrote the program. If you want your program to use more CPUs, you will have to perform more independant tasks in your program so it can utilise more CPUs.

Is there any way to tell Java to use all the available nodes to run a given program

Yes, but you have to do this in your code, otherwise it won't magically use more CPUs than your program tells it to.

JRE's fault I guess.

Either the JRE which was released 20 years ago has a bug which stops multi-threading working and no one has fixed it in that time even though about 10 million developers use it, or your program you just wrote which only one person uses has a bug in it. Which one sounds like it is more likely to have a bug?

If you want to see that you can use all the CPUs on your machine.

public class Main {
    public static void main(String[] args) {
        IntStream.range(0, 1000).parallel().forEach(i -> {
            long end = System.currentTimeMillis() + 1000;
            while (end > System.currentTimeMillis()) {
                // busy wait
            }
            System.out.println("task " + i + " finished");
        });
    }
}

This will use all the logical CPUs on your machine without any special options.

I have seen Java running on servers with 120 CPUs and 3 TB of main memory in production work just fine. You run into some problems when you have a Java program use multiple NUMA regions, but I doubt that is your issue here.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130