12

I do not understand why do I get the error KeyError: '[ 1351 1352 1353 ... 13500 13501 13502] not in index' when I run this code:

cv = KFold(n_splits=10)

for train_index, test_index in cv.split(X):
    f_train_X, f_valid_X = X[train_index], X[test_index]
    f_train_y, f_valid_y = y[train_index], y[test_index]

I use X (a Pandas dataframe) to split I cv.split(X).

X.shape
y.shape
Out: (13503, 17)
Out: (13503,)
seralouk
  • 30,938
  • 9
  • 118
  • 133
ScalaBoy
  • 3,254
  • 13
  • 46
  • 84

1 Answers1

29

The problem is the way you are trying to index the X using X[train_index]. You need to use .loc or .iloc since you have pandas dataframe.

Use this

cv = KFold(n_splits=10)

for train_index, test_index in cv.split(X):
    f_train_X, f_valid_X = X.iloc[train_index], X.iloc[test_index]
    f_train_y, f_valid_y = y.iloc[train_index], y.iloc[test_index]

1st way: Example using iloc

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))

df[[1,2]]
#KeyError: '[1 2] not in index'

df.iloc[[1,2]]
#    A   B   C   D
#1  25  97  78  74
#2   6  84  16  21

2nd way: Example by converting pandas to numpy in advance

df = df.values

#now this should work fine
df[[1,2]]
#array([[25, 97, 78, 74],
#      [ 6, 84, 16, 21]])
seralouk
  • 30,938
  • 9
  • 118
  • 133
  • Thank you so much! Does it mean that X[train_index] requires numpy array? – ScalaBoy Jun 28 '18 at 21:03
  • 1
    The default indexing of scikit-learn functions are expecting numpy arrays. Use .iloc or use `X = X.values` in advance. Please consider accepting my answer – seralouk Jun 28 '18 at 21:05