0

I'm encountering an intermittent error message in R after using parallel::dopar.

I use a dopar loop once after starting a fresh session and everything works. My script runs, takes a corpus, transforms it and outputs a matrix.

If I rm(list = ls()) and closeAllConnections() I am unable to do anything after that without hitting a memory error.

Tried to run the function again with slightly altered parameters gave "Error in mcfork() : unable to fork, possible reason: Cannot allocate memory" but then if I try to do ANYTHING else I get an error. I tried to type sessionInfo() which gave "Error in system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE) : cannot popen '/usr/bin/which 'uname' 2>/dev/null', probable reason 'Cannot allocate memory'"

I tried opening the shell in R studio (hosted R studio tools > shell) which gives a popup "cannot allocate memory".

I tried entering stopImplicitCluster() after my dopar loop as well as closeAllConnections()

I don't know where to look next? Does this sound familiar to anyone?

I noticed in the terminal top > 1 where I see each core that all my cores are at 100% sleeping but I'm not sure what that means. Here's a screen shot: enter image description here

Not sure what other information to provide?

This is the script that runs perfectly fine once in a fresh session, then seems to leave me with no memory.

clean_corpus <- function(corpus, n = 1000) { # n is length of each peice in parallel processing

  # split the corpus into pieces for looping to get around memory issues with transformation
  nr <- length(corpus)
  pieces <- split(corpus, rep(1:ceiling(nr/n), each=n, length.out=nr))
  lenp <- length(pieces)

  rm(corpus) # save memory

  # save pieces to rds files since not enough RAM
  tmpfile <- tempfile() 
  for (i in seq_len(lenp)) {
    saveRDS(pieces[[i]],
            paste0(tmpfile, i, ".rds"))
  }

  rm(pieces) # save memory since now these are saved in tmp rds files

  # doparallel
  registerDoParallel(cores = 12)
  pieces <- foreach(i = seq_len(lenp)) %dopar% {
    # update spelling
    piece <- readRDS(paste0(tmpfile, i, ".rds"))
    # spelling update based on lut
    piece <- tm_map(piece, function(i) stringi_spelling_update(i, spellingdoc))
    # regular transformations
    piece <- tm_map(piece, removeNumbers)
    piece <- tm_map(piece, content_transformer(removePunctuation), preserve_intra_word_dashes = T)
    piece <- tm_map(piece, content_transformer(function(x, ...) 
      qdap::rm_stopwords(x, stopwords = tm::stopwords("english"), separate = F)))
    saveRDS(piece, paste0(tmpfile, i, ".rds"))
    return(1) # hack to get dopar to forget the piece to save memory since now saved to rds
  } 

  stopImplicitCluster() # I added this but according to documentation I don;t think I need to since implicit clusters are closed automatically by doparallel?

  # combine the pieces back into one corpus
  corpus <- list()
  corpus <- foreach(i = seq_len(lenp)) %do% {
    corpus[[i]] <- readRDS(paste0(tmpfile, i, ".rds"))
  }
  corpus <- do.call(function(...) c(..., recursive = TRUE), corpus)
  return(corpus)

} # end clean_corpus function
Doug Fir
  • 19,971
  • 47
  • 169
  • 299
  • Kill the R processes. Then study the example in ´help("registerDoParallel")` and use it as a template for setting up and stoping the cluster. – Roland Aug 24 '17 at 10:22
  • You have no free swap left and most of RAM is in use. – Roland Aug 24 '17 at 10:24
  • Hi, on the screen shot how can you tell I have no free swap and most RAM is in use? Under %mem it says 52%. How do you know about swap from the screen? I'll try changing my code to match the example in ```helpdoparallel()``` but from reading their documentation I thought my approach was sound? – Doug Fir Aug 24 '17 at 10:29
  • Your main R process uses 52.3 % of memory. I assume there are other R processes (the workers). Try sorting by memory usage. I looked at the two lines after the CPU table (although I might interpret it wrong, I'm not an advanced linux user). – Roland Aug 24 '17 at 10:34
  • Looking at help("registerDoParallel") the example creates a cluster using ```cl <- makePSOCKcluster(2)```. Do I need a cluster though? I'm using hosted R studio on a single server? Are clusters diferent from cores? My understanding was that I just need to use multiple cores on a single machine. A posted about this previously. https://stackoverflow.com/questions/45779447/do-i-still-need-to-makecluster-if-im-already-doing-registerdoparallelcl/45799904?noredirect=1#comment78558415_45799904 – Doug Fir Aug 24 '17 at 10:49
  • On top of that, manually creating a cluster using makeCluster means you have to export all needed objects to the cluster, so for convenience is using just registerDoParallel() not simpler and just as efficient? I'm not sure – Doug Fir Aug 24 '17 at 10:52
  • A "cluster" can be on one computer. Also, the objects need to be exported anyway. The only difference is that you created the cluster implicitly and have consequently less control over it. – Roland Aug 24 '17 at 10:58
  • Ok thanks for the info but... what's the benefit of doing this if registerdoparallel() takes care of exporting and closing for you? – Doug Fir Aug 24 '17 at 10:59
  • Are you sure the child processes are closed? I suspect they aren't. Check that. – Roland Aug 24 '17 at 11:01
  • I don't know how to close them, I'm researching that now. My understanding was that doparallel woud take care of that but here it appears somehting is not working correctly – Doug Fir Aug 24 '17 at 11:03
  • https://www.tecmint.com/how-to-kill-a-process-in-linux/ – Roland Aug 24 '17 at 11:09
  • Yeah I already killed the rsession PID the problem I think is these child sleepers which are remaining and won't die – Doug Fir Aug 24 '17 at 11:11
  • Have you sorted the top output by mem usage? Killing the main process does not kill child processes (if they are indeed still there). – Roland Aug 24 '17 at 11:12
  • I did that just now. All show zero except every now and then PID 127591 uses 0.3% memory. I tried this command to see if I could kill it ```pkill -TERM -P 127591``` but the sleeping process are still there – Doug Fir Aug 24 '17 at 11:38
  • I might repost this question more specifically to doparallel leaving sleeping process'. What do you think? It might narrow it down a bit – Doug Fir Aug 24 '17 at 11:39
  • I'm not at all sure that you actually have sleeping R processes. – Roland Aug 24 '17 at 11:41
  • Oh hang on, I've misread the screen shot. I was looking at all the values of 100 which appear under the text "sleeping". But sleeping is not a column. These are 100% id... idle? – Doug Fir Aug 24 '17 at 11:47

0 Answers0