Slicing a Dask Dataframe

Question

I have the following code where I like to do a train/test split on a Dask dataframe

df = dd.read_csv(csv_filename, sep=',', encoding="latin-1",
                     names=cols, header=0, dtype='str')

But when I try to do slices like

for train, test in cv.split(X, y):
    df.fit(X[train], y[train])

it fails with the error

KeyError: '[11639 11641 11642 ..., 34997 34998 34999] not in index'

Any ideas?

Check if [this answer](https://stackoverflow.com/a/54507584/812102) helps. — Skippy le Grand Gourou, Feb 03 '19 at 21:08

MRocklin · Answer 1 · 2017-06-11T12:59:57.180

7

Dask.dataframe doesn't support row-wise slicing. It does support the loc operation if you have a sensible index.

However in your case of train/test splitting you will probably be better served by the random_split method.

train, test = df.random_split([0.80, 0.20])

You could also make many splits and concat in different ways

splits = df.random_split([0.20, 0.20, 0.20, 0.20, 0.20])

for i in range(5):
    trains = [splits[j] for j in range(5) if j != i]
    train = dd.concat(trains, axis=0)
    test = splits[i]

edited Jun 11 '17 at 12:59

answered Jun 10 '17 at 16:36

MRocklin

55,641
23
163
235

So basically I cannot do StratifiedKFold CV on a Dask dataframe? – Zubair Ahmed Jun 10 '17 at 17:35
1

You can't slice with normal means, but that doesn't mean that you can't do what you want. Note that `random_split` can accept a list of arbitrary size of fractions. – MRocklin Jun 10 '17 at 17:43
Any code sample for how can I achieve CV with this? – Zubair Ahmed Jun 11 '17 at 08:15
I've added a slightly longer example that should help you along – MRocklin Jun 11 '17 at 13:00
train = dd.concat(trains, axis=0) returns me 5 partitions instead of 1, when I do train['columnName'] similarly test['columnName'] also returns me 5 partitions of varying length what to do here? – Zubair Ahmed Jun 12 '17 at 19:47

Slicing a Dask Dataframe

1 Answers1

Linked