7

when i am trying to create a dask dataframe by reading a pickle file , iam getting an error

import dask.dataframe as dd
ds_df = dd.read_pickle("D:\test.pickle")

AttributeError: 'module' object has no attribute 'read_pickle'

but  it works fine with read_csv

And in pandas it was successful as usual.

So please correct me if i am doing something wrong there or in dask we can't create dataframe by reading a pickle file at all.

Satya
  • 5,470
  • 17
  • 47
  • 72

1 Answers1

12

Please note that dask.dataframe does not fully implement Pandas. You should not expect every pandas operations to have an analog in dask.dataframe.

We haven't chosen to implement reading from pickle files in particular because there is no way to read only part of a pickle file; everything gets dumped into memory at once. Because of this, pickle files don't have much value when it comes to reading large datasets piece by piece from disk.

If you're just looking for parallelism then I recommend using pandas.read_pickle along with dask.dataframe.from_pandas

df = pd.read_pickle(...)
ddf = dd.from_pandas(df, npartitions=8)
MRocklin
  • 55,641
  • 23
  • 163
  • 235
  • @Matthew-got it.can you please help me in http://stackoverflow.com/questions/34266263/reading-csv-with-separator-in-python-dask?noredirect=1#comment56274694_34266263 this. – Satya Dec 14 '15 at 17:20
  • i was compelled to try pickle reading bcoz of that csv reading with separator did not worked out. – Satya Dec 14 '15 at 17:23