0

I am trying to run this simple for loop as a parallel process as it requires lots of compute power. Any thoughts?

##Load files and libraries---
library(tidyverse)
library(caret)
library(insight)
library(MASS)
library(mfx)
library(furrr)

for (i in 1:16) {
      nested_inter$model[[i]]<- nb_thesis_inter(df= nested_nb$data[[i]], mdl= nested_nb$model[[i]])
      print (paste("Finished model ", i, "out of 16"))
    }

  #nested_inter<- nested_inter %>% 
  #  mutate(model= future_map2(.x= data, .y=model, .f = nb_thesis_inter))
Stat.Enthus
  • 335
  • 1
  • 12

2 Answers2

1

My go to is the future.apply package.

library(future.apply)
plan(multisession)

nested_inter$model = future_Map(nb_thesis_inter,
                                nested_nb$data,
                                nested_nb$model)

Two things to note.

  1. plan(multisession) allows Windows to be used in parallel. See ?plan for all options.
  2. I did not install all of the packages because the example was not reproducible. The future_Map call may need to be changed to future_map(function (x, y) nb_thesis_inter(df = x, mdl = y), ...) depending on the default argument order of nb_thesis_inter.
Cole
  • 11,130
  • 1
  • 9
  • 24
  • Thanks! It looks like it's running and working.. How can I get benchmarks on the runtime or set different parameters for it? I want to compare to the for loop – Stat.Enthus Oct 17 '20 at 12:11
  • ```system.time()```? I also like ```bench::mark()```. As far as more parameters, ```Map``` is ```Map(function(x, y) {your_function}, x_arg, y_arg)``` where we can keep adding more arguments if we want (e.g., we can have ```z_arg``` which would pass another argument to our function). In this example we already had a function ```nb_thesis_inter``` so we didn't have to create a new function within the ```Map``` call. – Cole Oct 17 '20 at 12:17
  • Oh. This is embarrassing - I'm not actually that good at this package! I think ```?plan``` which then links to various strategies is good. For example ```?multisession``` which Ied me to try ```plan(multisession, workers = 4L)``` which worked. – Cole Oct 17 '20 at 13:28
  • Thanks! I will try to post it in another thread. Right now this code seems right and not working :-) bnch_future<-bench::mark( nested_inter$model2 = future_Map(nb_thesis_inter, nested_nb$data, nested_nb$model)) – Stat.Enthus Oct 17 '20 at 18:53
0

you could use pmap:

nested_nb %>% furrr::future_pmap(function(...){
  row <- list(...)
  nb_thesis_inter(df = row$data, mdl= row$model)
})
Waldi
  • 39,242
  • 6
  • 30
  • 78