0

I frequently use the packages future.apply and future to parallelize tasks in R. This works perfectly well in my local machines. However, if I try to use them in a computer cluster, managed by PBS/TORQUE, the job gets killed for violating the resources policy. After reviewing the processes, I noticed that the resources_used.mem and resources_used.vmem as reported by qstat are ridiculously high. Is there any way to fix this?

Note: I already know and use the package batchtools and future.batchtools, but they produce jobs to launch to the queues, so this requires me to organize the scripts in a particular way, so I would like to avoid this for this specific example.

I have prepared the following MVE. As you can see, the code simply allocates a vector with 10^9 elements, and then performs, in parallel using future_lapply, some operations (here just a trivial check).

library(future.apply)

plan(multicore, workers = 12)

sample <- rnorm(n = 10^9, mean = 10, sd = 10)

print(object.size(sample)/(1024*1024)) # fills ~ 8 gb of RAM
options(future.globals.maxSize=+Inf) 
options(future.gc = TRUE)

future_lapply(future.seed = TRUE,
              X = 1:12, function(idx){
                # just do some stuff
                for(i in sample){
                  if (i > 0) dummy <- 1
                }
                return(dummy)
              })

If run on my local computer (no PBS-TORQUE involved), this works well (meaning no problem with the RAM) assuming 32Gb of RAM are available. However, if run through TORQUE/PBS on a machine that has enough resources, like this:

qsub -I -l mem=60Gb -l nodes=1:ppn=12 -l walltime=72:00:00

the job gets automatically killed due to violating the resources policy. I am pretty sure that this has to do with PBS/TORQUE not measuring correctly the resources used since, since if I check

qstat -f JOBNAME | grep used

I get:

resources_used.cput = 00:05:29
resources_used.mem = 102597484kb
resources_used.vmem = 213467760kb
resources_used.walltime = 00:02:06

Telling me that the process is using ~102Gb of mem and ~213Gb of vmem. It does not, you can actually monitor the node with e.g. htop and it is using the correct amount of RAM, but TORQUE/PBS is measuring much more.

D1X
  • 5,025
  • 5
  • 21
  • 36
  • Is PBS/Torque reports that, then that's how much the job consumes from its point of view. I'm pretty sure you can replicate this without futures, e.g. using 12 standalone R scripts. Note that each task ("iteration") uses `sample` and since you run 12 of those concurrently on that machine, you may end up with a minimal of 12*8*10^9/1024^3 GiB = 89.4 GiB. I've seen this happen even with forked processing, e.g. due to R's GC. Then add some extra memory overhead for each of the 12 R processes, it's not too far off to imagine you get the reported `mem` 102597484*1e3/1024^3 GiB = 95.6 GiB. – HenrikB Nov 16 '22 at 00:39
  • BTW, don't hard code `plan(multicore, workers = 12)`. It's sufficient to use `plan(multicore)`, because it will detect the PBS/Torque settings internally, cf. – HenrikB Nov 16 '22 at 00:40
  • @HenrikB But the actual memory used is not 89.4 GiB, since the biggest part of it is shared. Any way around this? Using different scripts is not an option. In this case it can be easily replicated, since it is a MVE, but not with the scripts I am dealing with. – D1X Nov 16 '22 at 08:06
  • @HenrikB Maybe not using futures but any other parallelization way? – D1X Nov 16 '22 at 08:33
  • @HenrikB Looking at the problem, I think that one option would then be to divide the share objects (in this case `sample`) and pass to each "worker" only the chunk that needs to work with. Is there a way to do this through `future.apply` without having to e.g. save them in a file and then load them inside each core? – D1X Nov 16 '22 at 10:00

0 Answers0