0

I have my data as ff arrays and stored perfectly on the disk, I'm trying to use parallel computation. Array's names are stored in the data frame DATA.wind in column Smoothed_name

DATA.wind<-subset(DATA,DATA$variable=="u"|DATA$variable=="v")
# Calculate the number of cores
no_cores <- detectCores() - 1
cl <- makeCluster(no_cores)
#Exporting functions and environment to cluster
clusterExport(cl, list("five_points_smoother","DATA.wind"),envir=globalenv())
 
parLapply(cl,
      1:length(DATA.wind$No),function(x){
        medium1<-five_points_smoother(get(DATA.wind$var_name[x]))
        assign(DATA.wind$Smoothed_name[x],ff(medium1,dim =dim(medium1))) 
      })

when i run parLapply i get this error:

`  Error in checkForRemoteErrors(val) : 
   one node produced an error: object 'u_1088632800' not found `

knowing that 'u_1088632800'is one of the arrays name stored in the df.

ahmathelte
  • 559
  • 3
  • 15
  • Maybe passing ``globalenv()`` to ``get(..., envir = globalenv())`` will work. Or export the frames through ``clusterExport``? – runr Jun 30 '20 at 00:30
  • @Nutle I have tried your solution if I understood it, however, no progress. here is my new loop : parLapply(cl, 1,function(x){ medium1<-get(DATA.wind$var_name[x],envir = globalenv()) clusterExport(cl,"medium1",envir=globalenv()) medium2<-five_points_smoother(medium1) assign(DATA.wind$Smoothed_name[x],ff(medium2,dim =dim(medium1))) }) – ahmathelte Jun 30 '20 at 15:57

0 Answers0