5

I have a database-like object containing many dask dataframes. I would like to work with the data, save it and reload it on the next day to continue the analysis.

Therefore, I tried saving dask dataframes (not computation results, just the "plan of computation" itself) using pickle. Apparently, it works (at least, if I unpickle the objects on the exact same machine) ... but are there some pitfalls?

Arco Bast
  • 3,595
  • 2
  • 26
  • 53
  • 1
    You can unpickle on any machine and get the same result. – grael Aug 25 '16 at 13:51
  • So is it enough to ensure, the data underlying the dataframes are accessible under the same path and then it should be safe? – Arco Bast Aug 25 '16 at 13:55
  • Yes, it should be safe. Pickle saves everything you need to restore the object to the same state as it was when it was pickled. – grael Aug 25 '16 at 14:06

1 Answers1

2

Generally speaking it is usually safe. However there are a few caveats:

  1. If your dask.dataframe contains custom functions, such as with with df.apply(lambda x: x) then the internal function will not be pickleable. However it will still be serializable with cloudpickle
  2. If your dask.dataframe contains references to files that are only valid on your local computer then, while it will still be serializable the re-serialized version on another machine may no longer be useful
  3. If your dask.dataframe contains dask.distributed Future objects, such as would occur if you use Executor.persist on a cluster then these are not currently serializable.
  4. I recommend using a version >= 0.11.0.
MRocklin
  • 55,641
  • 23
  • 163
  • 235
  • We have the same use case. Would like to share computations between data scientist within a company. The architecture we are thinking about is pickling the dask dataframes (plan of execution) and storing it pickled in a key value store. It should be executable on a different cluster at a later time. Is point 3 still the case, and if so, could you mentor me towards a PR on this? – Paul-Armand Verhaegen Oct 25 '18 at 09:54