in R, how to make each dataframe generated in foreach loops available in global environment

Question

In R, how to make each dataframe generated in foreach loop available as individual dataframes in global environment

I was only able to save them into a list (x), but the list is of 3 layers; there are over 40,000 dataframes, and unpacking them is very time consuming.

x <- foreach(q=1:countq, .export = ls(globalenv())) %do% {
        foreach(p=1:countp, .export = ls(globalenv())) %do% {
          foreach(o=1:countero, .export = ls(globalenv())) %dopar% {
            n<-rbind(df_o, df_p, df_q)
          }
        }

It would be nice to have dataframes n1, n2, n3, ... till n40000 from this nested foreach loop.

You won't get far trying to access variables in one R session's environment from another R session. And while I understand the difficulty of dealing with multi-tier lists-of-lists, I suggest your question should have less (nothing?) to do with `foreach` and all about "nested-lists". Perhaps if you provide an example (maybe 2-3 simple triple-nested lists), we can help with the aggregation and/or extraction you need. (For that, we'll also need to know what you need from them, i.e., expected output.) — r2evans, Jul 09 '19 at 23:08
For a good sample of data, I suggest using the output of `dput`, perhaps something like `mt <- mtcars[1:2,]; dput(replicate(3, replicate(2, replicate(2, mt, simplify=F), simplify=F), simplify=FALSE))`. Just a thought, the key is `dput(head(x))` where `x` is your nested-list. — r2evans, Jul 09 '19 at 23:15

score 0 · Answer 1 · answered Jul 10 '19 at 02:15

Your data should be easy to use once you convert your list of lists of lists of data.frames into a single data.frame, e.g., with data.table::rbindlist.

library(doParallel)
registerDoParallel( cores = 2 )
countq <- countp <- countero <- 30
d <- mtcars
x <- 
foreach(q=1:countq) %do% {
    foreach(p=1:countp) %do% {
        foreach(o=1:countero) %dopar% {
            data.frame( q=q, p=p, o=o, d )
        }
    }
}
x <- lapply( x, function(u) lapply(u, data.table::rbindlist) )
x <- lapply( x, data.table::rbindlist )
x <- data.table::rbindlist(x)
x <- as.data.frame(x)

in R, how to make each dataframe generated in foreach loops available in global environment

1 Answers1