1

I have an 8-core RHEL Linux machine running R 4.0.2.

If I ask R for the number of cores, I can confirm that 8 are available.

> print(future::availableWorkers())

[1] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
[7] "localhost" "localhost"

> print(parallel::detectCores())

[1] 8

However, if I run this simple example

f <- function(out=0) {
    for (i in 1:1e10) out <- out + 1
}

output <- parallel::mclapply(1:8, f, mc.cores = 8)

my top indicates that only 1 core is being used (so that each worker is using 1/8th of that core, or 1/64th of the entire machine).

%Cpu0  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  2.0 us,  0.0 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32684632 total, 28211076 free,  2409992 used,  2063564 buff/cache
KiB Swap: 16449532 total, 11475052 free,  4974480 used. 29213180 avail Mem

  PID USER  PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 3483 user  20   0  493716  57980    948 R  1.8  0.2   0:18.09 R
 3479 user  20   0  493716  57980    948 R  1.5  0.2   0:18.09 R
 3480 user  20   0  493716  57980    948 R  1.5  0.2   0:18.08 R
 3481 user  20   0  493716  57980    948 R  1.5  0.2   0:18.09 R
 3482 user  20   0  493716  57980    948 R  1.5  0.2   0:18.09 R
 3484 user  20   0  493716  57980    948 R  1.5  0.2   0:18.09 R
 3485 user  20   0  493716  57980    948 R  1.5  0.2   0:18.09 R
 3486 user  20   0  493716  57980    948 R  1.5  0.2   0:18.09 R

Does anyone know what might be going on here? Another StackOverflow question that documents similar behavior is here. It's clear that I messed up the install somehow. I followed these install instructions for RHEL 7. I'm guessing there is a dependency missing, but I have no idea where to look. If anyone has any ideas of diagnostics to run, etc., they would be most appreciated.

For further context, I have R 3.4.1 also installed on my machine, and when I run this code, everything works fine. (I installed that version through yum.)

I also installed R 4.0.3 yesterday using the same instructions linked above, and it suffers from the same problem.

Tyler R.
  • 461
  • 6
  • 15
  • 1
    Do you get the same result with `workers <- parallel::makeCluster(8, type = "PSOCK"); parallel::parLapply(1:8, f, cl = workers)`? – jared_mamrot Jan 12 '21 at 23:02
  • @jared_mamrot Yes, I get identical behavior using the code you suggested. – Tyler R. Jan 13 '21 at 17:34
  • just tested it on a Linux / Debian virtual machine with two cores allocated : both cores are activated 100%. – Waldi Jan 13 '21 at 21:59
  • 1
    If you're having the same issue with a 'socket' approach, it may be an issue with openBLAS. Try the troubleshooting suggestion (`system(sprintf("taskset -p 0xffffffff %d", Sys.getpid()))`) and solution here: https://stackoverflow.com/a/13127132/12957340 / https://stackoverflow.com/questions/12924698/parallel-processing-in-r-limited – jared_mamrot Jan 13 '21 at 22:18
  • Thanks @jared_mamrot. Here is the output I see from your suggestion: `> system(sprintf("taskset -p 0xffffffff %d", Sys.getpid()))` `pid 10256's current affinity mask: ff` `pid 10256's new affinity mask: ff` So it doesn't seem to be the problem. – Tyler R. Jan 14 '21 at 13:28
  • 1
    So running `system(sprintf("taskset -p 0xffffffff %d", Sys.getpid()))` then running your simple example still only uses one core... Can you please include the output from the command `sessionInfo()` in your question? Also it might be worth [installing R from source](https://docs.rstudio.com/resources/install-r-source/) – jared_mamrot Jan 14 '21 at 22:26
  • @jared_mamrot I ended up installing R 4.0.3 this afternoon, and the solution you referenced fixes my issue. If you wouldn't mind adding this as an answer, I'll be happy to award you the bounty. Thanks so much for your help! – Tyler R. Jan 15 '21 at 01:17
  • Fantastic! Glad it worked :) – jared_mamrot Jan 15 '21 at 02:23

1 Answers1

1

First run

system(sprintf("taskset -p 0xffffffff %d", Sys.getpid()))

then your simple example

f <- function(out=0) { for (i in 1:1e10) out <- out + 1 }
output <- parallel::mclapply(1:8, f, mc.cores = 8)

works on all 8 cores.

Tyler R.
  • 461
  • 6
  • 15
jared_mamrot
  • 22,354
  • 4
  • 21
  • 46
  • 1
    And an even better solution is to add the `system()` call above to my `.Rprofile` file in my home directory, inside the function `.First()` as explained here: https://stackoverflow.com/questions/33054164/is-it-possible-to-execute-a-command-automatically-upon-launch-in-r. This ensures that the command gets called at the beginning of every R session. – Tyler R. Jan 16 '21 at 13:10