How to clusterExport in a foreach loop (OS:Windows)

Question

I have the following problem. I need the timeindex of variables in my Global Environment, but when I want to export them into my cluster during the parallel processing from my Global Environment, I'm getting the following message:

Error in { : task 1 failed - "object 'Szenario' not found"

A minimal example of my original code, which produces the error:

    Historical <- structure(c(18.5501872568473, 24.3295787432955, 14.9342384460288, 
    13.0653757599636, 8.67294618896797, 13.4587662721594, 20.792126254714, 
    17.5162747884424, 28.8253151239752, 23.0568765432192), index = structure(c(-7305, 
    -7304, -7303, -7302, -7301, -7300, -7299, -7298, -7297, -7296
    ), class = "Date"), class = "zoo")

    Szenario <- structure(c(10.2800684124652, 14.5495871489361, 9.8565852930294, 
    21.1654540678686, 21.1936990312861, 12.4209005842752, 9.77473132000267, 
    17.1997402736739, 17.884107611858, 13.622588360244), index = structure(c(13149, 
    13150, 13151, 13152, 13153, 13154, 13155, 13156, 13157, 13158
    ), class = "Date"), class = "zoo")

    library(doParallel)   
    library(foreach)     
    library(raster)
    library(zoo)
    library(parallel)

    # Parallelisation Settings
    # Definition of how many cores you want to use
    UseCores <- detectCores() -2 # -1 at max because one core has to be used for other tasks 
    # Register CoreCluster
    cl <- makeCluster(UseCores)
    registerDoParallel(cl)

foreach(fn=1:1) %dopar% {

  library(raster) # needed libraries have to be loaded inside the loop, while parallel processing occurs
  library(zoo)
  library(base)
  library(parallel)

  #In my original script, I'm looping through Filenames, which are called like my variables in my Global environment (without .tif at the end of the filename), variables names are saved as characters
  file.referenz.name <- c("Historical")
  file.szenario.name <- c("Szenario")

  #Create timeindex für rasterstacks to subset later on with them (getZ, setZ)

  clusterExport(cl, varlist = c(file.szenario.name, file.referenz.name), envir = .GlobalEnv)
  time.index.szenario <- index(get(file.szenario.name))
  time.index.referenz <- index(get(file.referenz.name))

}

#end cluster
stopCluster(cl)

score 1 · Accepted Answer · answered Aug 27 '17 at 01:42

1

Try this

foreach(fn=1:1, .export=c("Szenario"), 
                .packages=c("raster", "zoo", "base", "parallel") %dopar%

And it's confusing to clusterExport inside a %dopar% {}. You can either clusterExport to each cl before foreach, or simply use .export in foreach

You can remove library statements in the %dopar% {}.

answered Aug 27 '17 at 01:42

CPak

13,260
3
30
48

Thank you. Can I export all my variables from my global environment? .Like this maybe: `.export=environment()` – Nucore Aug 27 '17 at 01:54
1

Okay, its `.export=ls(.GlobalEnv)` – Nucore Aug 27 '17 at 02:06
1

Be careful exporting *everything*, because exporting many and/or large objects not needed to be exported will add a substantial amount of overhead. Instead, try to carefully identify the set of globals needed, e.g. via trial-and-error. Alternatively, you can use the doFuture backend, cf. https://stackoverflow.com/a/45898676/1072091 – HenrikB Aug 27 '17 at 03:39
Thank you for the advise! I will take note of this. – Nucore Aug 30 '17 at 16:39

How to clusterExport in a foreach loop (OS:Windows)

1 Answers1