0

I have a sizeable set (n ~= 20) of static targets of which I need to load a large number of different subsets for different analyses. Currently, I'm using an object loading function that wraps tar_read_raw for this, but this breaks my pipeline's ability to track upstream changes for any code that depends on this object loading function (right?). E.g.:

load_objects <- function(names = c('A', 'B')) {
  purrr::map(names, ~targets::tar_read_raw(glue::glue('obj{.x}')))
}

Should I specify any potentially relevant subsets into new targets using tar_combine or is there a more flexible way that doesn't require me to define new targets upfront while also allowing targets to track changes to the upstream static targets?

Thanks :)

Extrapolator
  • 343
  • 3
  • 7
  • 2
    It is almost always preferable to reference targets in commands and/or use tar_combine() rather than an explicit tar_read() call inside a target. That way, the package understands the right dependency relationships, your code is cleaner, and the package makes its own decisions about how to manage memory and storage. I would say have a look at https://github.com/ropensci/tarchetypes/discussions/82. – landau Mar 10 '22 at 12:36
  • Thanks, the example there looks very elegant. I adapted it to my use case but noticed `tar_combine` writes the combined objects to disk, whereas I was hoping it would write out recipes/references to its upstream targets, to be rerun when needed. This current way won't scale for me. Can I have referenced objects somehow? Thanks – Extrapolator Mar 11 '22 at 10:06
  • Would you prefer me to ask questions over at github? – Extrapolator Mar 11 '22 at 10:07
  • Perhaps we could have another value to the 'format' argument, which would specify that the target is merely a recipe and should get lazily evaluated every time its value is needed and that the value should not be written to the objects folder? – Extrapolator Mar 11 '22 at 10:23
  • Never mind: I think I can forego combining targets beforehand. I'd migrate the downstream targets that take combinations of targets into the `tar_map` that was making the combinations, obviating the need for the intermediate `tar_combine` steps. Duh! – Extrapolator Mar 11 '22 at 10:38

0 Answers0