1

I successfully ran a complex workflow on a remote computer. On there I can check that all is up to date and explore the results without problem.

I would like to move the .drake folder to my local computer so I can explore the results more easily.

When I do that (through copying and pasting the folder across machines), I face the problem that the local session thinks that all my target are outdated. The .drake folder pointed to in drake_cache() in the local computer is the correct one, so I don't understand what's going on.

N'Bayramberdiyev
  • 5,936
  • 7
  • 27
  • 47
Alexandre Courtiol
  • 71
  • 4
  • I have a very similar issue. I have run part of my plan in a cluster, but now I want to go back to my computer. Result: some parts are outdated. Some input files are missing (they are too heavy for my computer, thats why I keep them in the cluster...). So I'm using `readd()` to read the targets, as Alexandre suggests. Then I am creating a new plan for the "local" part of my project. Anyway, I would love `drake` to be able to handle this in a more elegant manner. Maybe next versions will, but anyway it is a difficult task in my opinion. – Javi_VM Jul 02 '20 at 08:33

1 Answers1

0

I do not think I know enough at this point to confidently offer an explanation. Is it possible to share a small reproducible example so we can investigate?

When you copy a drake workflow from one machine to another, it is usually not enough to copy the .drake folder if you want the targets to stay up to date. I strongly recommend you copy your entire project, including code files (and file_in()/file_out()/knitr_in() files if you declared any). I know it may seem counterintuitive, but the status of each target depends on the functions and other objects loaded into your R session at the time, so the code that sources supporting scripts and calls make() also needs to be copied. I recommend having a look at https://books.ropensci.org/drake/projects.html, especially https://books.ropensci.org/drake/projects.html#safer-interactivity. You may have better luck with r_make() than regular make().

If you use namespaced function calls (e.g. pkg::fun()) then you should also make sure your package environments agree between machines. In fact, both machines should probably have the same version of R and the same package environments anyway. renv is great for locking down the package environment of a project.

For even more portability, containerization could really help. Example: https://gitlab.com/ecohealthalliance/drake-gitlab-docker-example.

landau
  • 5,636
  • 1
  • 22
  • 50
  • I did use r_make() but indeed I am not using the same R + pkg versions (the remote is outdated and there is little I can do about that). Isn't there a way though to make drake happy enough so I can at least explore my objects locally? – Alexandre Courtiol Jan 19 '20 at 14:06
  • If all you want to do is explore your targets, it does not actually matter whether or not they are outdated. In fact, you can explore historical values of those targets if you did not run garbage collection on the cache: https://books.ropensci.org/drake/walkthrough.html#history-and-provenance – landau Jan 19 '20 at 14:21
  • Unfortunately, I had tried already but ```r_make()``` does not seem to have saved the history: ```Error: no history. Call make(history = TRUE) next time```. No worries, I will rerun everything locally... – Alexandre Courtiol Jan 19 '20 at 15:20
  • Even without history, you can `loadd()` and `readd()` current existing targets to informally explore stuff in your R session. That part doesn't need `make()` or `r_make()`. No need for history either, I just thought it was worth a mention before. – landau Jan 19 '20 at 15:28
  • Unfortunately, the keys are not found after moving the folder so that does not work either. Yet the ```drake_cache()``` info are correct and the sha1sum identical between the original and the cloned ```.drake``` folders... I stop now because we are branching off usual stackoverflow short answers... I need a RME – Alexandre Courtiol Jan 19 '20 at 17:03
  • That's what I love about GitHub issues. I prefer it when people post reprexes to https://github.com/ropensci/drake/issues. – landau Jan 19 '20 at 20:31