Load many feather files in a folder into dask

Question

With a folder with many .feather files, I would like to load all of them into dask in python.

So far, I have tried the following sourced from a similar question on GitHub https://github.com/dask/dask/issues/1277

files = [...]
dfs = [dask.delayed(feather.read_dataframe)(f) for f in files]
df = dd.concat(dfs)

Unfortunately, this gives me the error TypeError: Truth of Delayed objects is not supported which is mentioned there, but a workaround is not clear.

Is it possible to do the above in dask?

You need to turn your delayed object into an actual dataframe. Try `dd.from_delayed()`. — CJR, Aug 08 '19 at 01:35

score 8 · Accepted Answer · answered Aug 08 '19 at 19:10

Instead of concat, which operates on dataframes, you want to use from_delayed, which turns a list of delayed objects, each of which represents a dataframe, into a single logical dataframe

dfs = [dask.delayed(feather.read_dataframe)(f) for f in files]
df = dd.from_delayed(dfs)

If possible, you should also supply the meta= (a zero-length dataframe, describing the columns, index and dtypes) and divisions= (the boundary values of the index along the partitions) kwargs.

Load many feather files in a folder into dask

1 Answers1

Linked