Workflow for drake plan with increasing input

Question

I have a drake plan which uses a input folder with file_in. Then reads each file inside the folder and makes a number of transformations. Finally, it joins the results.

If I add a new file, I would like that the new calculations in plan are only applied to this file, and then joined to the previous results. However, what the plan does is: it detects a change in target, then recalculates all targets based on that target.

Note: The number of files is quite large (several thousands), and calculations heavy.

Solution (look at landau's solution for a better solution)

This solution completes the answer I marked as solution:

Any file or directory you declare with file_in() or target(format = "file") is treated as an irreducible unit of data, and this behavior in drake will not change in future development. But you can split up the files among multiple targets so some targets stay up to date if a file changes.

library(drake)

drake_plan(
  input  = target(list.files(file_in("/path/to/folder")),format="file"),
  target1 = target(do_stuff1(input), dynamic=map(input)) 
)

This will make dynamic targets, and therefore the new files will create new dynamic targets, but the old target will not be re-calculated.

landau · Accepted Answer · 2020-09-17T13:17:15.420

Any file or directory you declare with file_in() or target(format = "file") is treated as an irreducible unit of data, and this behavior in drake will not change in future development. But you can split up the files among multiple targets so some targets stay up to date if a file changes.

library(drake)
group1 <- c("file1", "file2")
group2 <- c("file3", "file4")
drake_plan(
  target1 = do_stuff(file_in(!!group1)),
  target2 = do_stuff(file_in(!!group2))
)
#> # A tibble: 2 x 2
#>   target  command                               
#>   <chr>   <expr_lst>                            
#> 1 target1 do_stuff(file_in(c("file1", "file2")))
#> 2 target2 do_stuff(file_in(c("file3", "file4")))

^{Created on 2020-09-04 by the reprex package (v0.3.0)}

With dynamic branching

Dynamic branching over files is trickier, and file_in() is for static targets only. Even then, it may be suboptimal to create a dynamic sub-target for every single file because you have thousands of them. It is probably better to batch files into groups and give each group to a sub-target. But if you still want to dynamically branch over every single file, here is the way to do it that ensures each file is properly reproducibly tracked for changes.

library(drake)
drake_plan(
  # Always run to get the latest set of file paths.
  untracked_files = target(
    list.files("directory_with_files", full.names = TRUE),
    trigger = trigger(condition = TRUE)
  ),
  # Map over the vector of file paths and reproducibly track each one.
  tracked_files = target(
    untracked_files,
    dynamic = map(untracked_files)
  ),
  # Map over the tracked files and analyze each one.
  analyses = target(
    do_stuff(tracked_files),
    dynamic = map(tracked_files)
  )
)
#> # A tibble: 3 x 4
#>   target       command                          trigger           dynamic       
#>   <chr>        <expr_lst>                       <expr_lst>        <expr_lst>    
#> 1 untracked_f… list.files("directory_with_file… trigger(conditio… NA           …
#> 2 tracked_fil… untracked_files                … NA              … map(untracked…
#> 3 analyses     do_stuff(tracked_files)        … NA              … map(tracked_f…

^{Created on 2020-09-17 by the reprex package (v0.3.0)}

This is slightly easier in targets due to tarchetypes::tar_files().

I reached a solution based on your answer, so I'm editing your answer and accepting it as "the answer". Thank you! — Javi_VM, Sep 07 '20 at 07:53
My edit does not seem to be accepted, so I wrote it in my "question" post — Javi_VM, Sep 17 '20 at 07:43
Ah, I see you want to use dynamic branching for this. Please see the edit to my answer. It requires a bit more work to dynamically map over files in a way that tracks them correctly. And because you have thousands, I recommend batching anyway (i.e. multiple files per target/sub-target) so you don't end up with thousands of targets. — landau, Sep 17 '20 at 13:19
To be clear, with thousands of targets, you are likely to incur overhead. It is usually best to have only a couple hundred targets for optimal performance. — landau, Sep 17 '20 at 13:25

Workflow for drake plan with increasing input

Solution (look at landau's solution for a better solution)

1 Answers1

With dynamic branching