I have 2 datasets and applying 5 different ML models.
Dataset 1:
def dataset_1():
...
...
bike_data_hours = bike_data_hours[:500]
X = bike_data_hours.iloc[:, :-1].values
y = bike_data_hours.iloc[:, -1].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
return X_train, X_test, y_train.reshape(-1, 1), y_test.reshape(-1, 1)
The shape is (400, 14) (100, 14) (400, 1) (100, 1). The dtypes: object (int64, float64).
Dataset 2:
def dataset_2():
...
...
final_movie_df = final_movie_df[:500]
X = final_movie_df.iloc[:, :-1]
y = final_movie_df.iloc[:, -1]
gs = GroupShuffleSplit(n_splits=2, test_size=0.2)
train_ix, test_ix = next(gs.split(X, y, groups=X.UserID))
X_train = X.iloc[train_ix]
y_train = y.iloc[train_ix]
X_test = X.iloc[test_ix]
y_test = y.iloc[test_ix]
return X_train.shape, X_test.shape, y_train.values.reshape(-1,1).shape, y_test.values.reshape(-1,1).shape
The shape is (400, 25) (100, 25) (400, 1) (100, 1). The dtypes: object (int64, float64).
I am using different models. The code is
X_train, X_test, y_train, y_test = dataset
fold_residuals, fold_dfs = [], []
kf = KFold(n_splits=k, shuffle=True)
for train_index, _ in kf.split(X_train):
if reg_name == "RF" or reg_name == "SVR":
preds = regressor.fit(X_train[train_index], y_train[train_index].ravel()).predict(X_test)
elif reg_name == "Knn-5":
preds = regressor.fit(X_train[train_index], np.ravel(y_train[train_index], order="C")).predict(X_test)
else:
preds = regressor.fit(X_train[train_index], y_train[train_index]).predict(X_test)
But I am getting a common error like this, this, and this. I have gone through all of these posts, but getting no idea about the error. I have used iloc and values given as a solution for the visited links.
preds = regressor.fit(X_train[train_index], y_train[train_index]).predict(X_test)
File "/home/fgd/.local/lib/python3.8/site-packages/pandas/core/frame.py", line 3030, in __getitem__
indexer = self.loc._get_listlike_indexer(key, axis=1, raise_missing=True)[1]
File "/home/fgd/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1266, in _get_listlike_indexer
self._validate_read_indexer(keyarr, indexer, axis, raise_missing=raise_missing)
File "/home/fgd/.local/lib/python3.8/site-packages/pandas/core/indexing.py", line 1308, in _validate_read_indexer
raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Int64Index([ 0, 1, 3, 4, 5, 6, 7, 9, 10, 11,\n ...\n 387, 388, 389, 390, 391, 392, 393, 395, 397, 399],\n dtype='int64', length=320)] are in the [columns]"
Here, if I use train_test_split instead of GroupShuffleSplit then the code is working. However, I want to use GroupShuffleSplit based on the UserID so that the same user does not split for both train and test. Could you tell me how can I solve the problem while I will use GroupShuffleSplit?
Could you tell me why I am getting the error for dataset_2 while dataset_1 is working fully fine (and the shape and dtypes) are the same for both datasets.
