1

So, I am running on a Linux cluster with lots of compute nodes to choose from. I get exclusive use of the node. Batch submissions. I am running into issues limiting the number of threads. I should mention I have a parfor loop. When I start matlab with the -singleCompThread option, it doesn't seem to work.

I submitted to a node with 2 cores, used the mentioned single thread option, and when I check on the submitted job, it starts running with 32 threads. Once it hits the parfor loop, it has over 600 threads.

I also want to run with a specific amount of multiple threads, not a single thread. So, I use the function maxNumCompThreads(32); to limit to 32 threads. I put this function at the beginning of the file, and also right inside the parfor loop. I then submit the job to a node with 32 cores. It uses way more than 32 threads.

Any guidance? I have no idea.

whoami
  • 85
  • 1
  • 7
  • “so many threads still spawn and cause thrashing” Thrashing is caused by excessive memory usage, not excessive thread usage. Threads share memory. Are you maybe talking about MATLAB processes, rather than threads? Are you spawning MATLAB workers in your `parfor` loop? Or are you setting it up to do multithreading instead? – Cris Luengo Feb 02 '21 at 19:35
  • You are right, no thrashing. I am setting it up to do multithreading, I don't spawn workers inside the parfor. – whoami Feb 02 '21 at 19:54
  • And you set it up to do 32 threads when you have only two cores? – Cris Luengo Feb 02 '21 at 20:49
  • Oh no, in that case I put it on a node with 32 cores.Ok, I am in the interactive session now, matlab running, pulled up htop. I am on a 16 core machine, set matlab to use maxCompThreads(16), and set parfor to only use a maximum of 16 threads. The node says only one thread is running. But I see much more that 16 Matlab processes in htop, and only 1 thread running, sometimes – whoami Feb 02 '21 at 21:05
  • More than 16 MATLAB processes? I only see one on my machine, and AFAIK there shouldn't be more (though I might be outdated on that concept?). If you see multiple processes, you might either be (1) starting many MATLAB jobs at the same time, or (2) configuring your parallel pool to have that many worker processes. – Cris Luengo Feb 02 '21 at 21:09
  • Matlab log is claiming 12 workers. Shrug. This seems like a bit of a dead end so I'm going to keep troubleshooting solo. I really appreciate the input. It helped. – whoami Feb 02 '21 at 22:13

1 Answers1

0

It may well be that you're seeing non-computational threads being used by MATLAB. You could try launching MATLAB in -nojvm mode - that will further reduce the number of non-computational threads that get launched. Unfortunately that mode isn't compatible with parfor. Also, which version of MATLAB are you using? (There used to be problems with older releases spawning lots of JVM garbage collection threads in higher core count machines)

Edric
  • 23,676
  • 2
  • 38
  • 40
  • 2018b. Ok thanks, I will have to go to an interactive node in a bit and look at the thread count there. – whoami Feb 02 '21 at 19:47
  • Ok, I am in the interactive session now, matlab running, pulled up htop. I am on a 16 core machine, set matlab to use maxCompThreads(16), and set parfor to only use a maximum of 16 threads. The node says only one thread is running (so you were right, there are many threads just not being used). But I see much more that 16 Matlab processes in htop, and only 1 thread running. – whoami Feb 02 '21 at 21:04
  • In that scenario, you should have got 16 computational threads associated with the "main" MATLAB process, and one computational thread each for the 16 worker MATLABS (unless you changed the settings to make those worker processes run multithreaded). That ought to give you 17 total MATLAB processes as reported by `ps -C MATLAB`. – Edric Feb 03 '21 at 07:20