11

I am trying to take advantage of a quad-core machine by parallelizing a costly operation that is performed on a list of about 1000 items.

I am using R's parallel::mclapply function currently:

res = rbind.fill(parallel::mclapply(lst, fun, mc.cores=3, mc.preschedule=T))

Which works. Problem is, any additional subprocess that is spawned has to allocate a large chunk of memory:

enter image description here

Ideally, I would like each core to access shared memory from the parent R process, so that as I increase the number of cores used in mclapply, I don't hit RAM limitations before core limitations.

I'm currently at a loss on how to debug this issue. All of the large data structures that each process accesses are globals (currently). Is that somehow the issue?

I did increase my shared memory max setting for the OS to 20 GB (available RAM):

$ cat /etc/sysctl.conf 
kern.sysv.shmmax=21474836480
kern.sysv.shmall=5242880
kern.sysv.shmmin=1
kern.sysv.shmmni=32
kern.sysv.shmseg=8
kern.maxprocperuid=512
kern.maxproc=2048

I thought that would fix things, but the issue still occurs.

Any other ideas?

Clayton Stanley
  • 7,513
  • 9
  • 32
  • 46
  • 1
    There is always the solution to throw in more computing power. Personally, I would try to run the job on one of the free cluster instance of the [EBI R-cloud](http://www.ebi.ac.uk/Tools/rcloud). They have nice servers and its free. Looking at the time you could spend modifying the parallel function I think its worth a shot. Otherwise launch an Amazon instance with tons of RAM and the BioConductor AMI. – David R. Feb 02 '13 at 11:19
  • 2
    Any luck sorting this out or understanding it further? I'm dealing with something similar, http://stackoverflow.com/q/20430773/210673 – Aaron left Stack Overflow Dec 07 '13 at 03:22
  • 1
    @Aaron No progress yet, as I haven't needed multicore for calculations recently. I may have to revisit this in a few months. If I figure out anything new, I'll make sure to update the post. – Clayton Stanley Dec 07 '13 at 22:36
  • 1
    Thanks! If I make any progress it will be noted on the post I linked to. – Aaron left Stack Overflow Dec 09 '13 at 03:22
  • 1
    @Aaron Completely new dataset this time around, I'm not doing anything strange with the large globals like a copy or modify, using newest R binary and included 'Parallel' package, and mclapply, and the same behavior still occurs. – Clayton Stanley May 16 '14 at 19:27

3 Answers3

8

Just the tip what might have been going on R-devel Digest, Vol 149, Issue 22

Radford Neal's answer from Jul 26, 2015:

When mclapply forks to start a new process, the memory is initially shared with the parent process. However, a memory page has to be copied whenever either process writes to it. Unfortunately, R's garbage collector writes to each object to mark and unmark it whenever a full garbage collection is done, so it's quite possible that every R object will be duplicated in each process, even though many of them are not actually changed (from the point of view of the R programs).

Dead Vil
  • 450
  • 4
  • 11
  • 1
    While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. But anyway, your answer is more than appreciated. Thank you :) – koceeng Mar 20 '17 at 08:59
5

Linux and macosx have a copy-on-write mechanism when forking, it means that the pages of memory are not actually copied, but shared until the first write. mclapply is based on fork(), so probably (unless you write to your big shared data), the memory that you see reported by your process lister is not actual memory.

But, when collecting the results, the master process will have to allocate memory for each returned result of the mclapply.

To help you further, we would need to know more about your fun function.

Karl Forner
  • 4,175
  • 25
  • 32
1

I guess I would have thought this would not have used extra memory because of the copy-on-write functionality. I take it the elements of the list are large? Perhaps when R passes the elements to fun() it is actually making a copy of the list item instead of using copy on write. If so, the following might work better:

fun <- function(itemNumber){
  myitem <- lst[[itemNumber]]
  # now do your computations
}
res = rbind.fill(parallel::mclapply(1:length(lst), fun, mc.cores=3, mc.preschedule=T))

Or use lst[[itemNumber]] directly in your function. If R/Linux/macos isn't smart enough to use copy-on-write as you wrote the function, it may with this modified approach.

Edit: I assume you are not modifying the items in the list. If you do, R is going to make copies of the data.

farnsy
  • 2,282
  • 19
  • 22