Using validation set for testing the generalization power of model

Question

I had a question regarding hyperparamter tuning and looking for best fitted model (looking for the best fitted model for a particular dataset). I got recommend that I should split my data into three sets, rather than two (training and testing only):

_Training

_Validation

_Testing

and use grid search (cross validation) on my training set, after grid search (cross validation), I may use another set "validation set" for testing the generalization power of my model (performance on unseen data), I may change some parameter after that. However, I do not know how to use validation set for testing the generalization power of my model.

My Code:

dt = DecisionTreeClassifier(random_state=12)
max_depth = [int(d) for d in np.linspace(1,20,20)]
max_features = ['log2', 'sqrt','auto']
criterion = ['gini', 'entropy']
min_samples_split = [2, 3, 50, 100]
min_samples_leaf = [1, 5, 8, 10]
grid_param_dt = dict(max_depth=max_depth, max_features=max_features, min_samples_split=min_samples_split, min_samples_leaf=min_samples_leaf, criterion=criterion)
gd_sr_dt = GridSearchCV(estimator=dt, param_grid=grid_param_dt, scoring='accuracy', cv=10)

gd_sr_dt.fit(x_train, y_train)
best_parameters_dt = gd_sr_dt.best_params_
print(best_parameters_dt)

and I get the hyperparameter tuning as below:

{'criterion': 'gini', 'max_depth': 9, 'max_features': 'log2', 'min_samples_leaf': 10, 'min_samples_split': 50}

How to use the validation set for testing the generalization power of model with these hyperparameters?

score 0 · Accepted Answer · answered Jan 06 '21 at 08:47

This question is still not clear. However, if you mean that you need to have a validation dataset, you are already doing that. With each combination of possible parameters, you are creating 10 folds (through GridSearchCV), that means 10 out-of-fold datasets, which are basically used to assess the generalization capability of the model using selected parameters on unseen data.

If you mean that you need to have another dataset and validate the model against, I personally follow one of the two following options:

Getting the best model with the lowest error in a fold: by selecting model=gd_sr_dt.base_estimator_ and then testing the validation dataset preds=model.predict(X_val).
Taking an ensemble of the models that revealed the best performance: By that, you will take the average of the predictions of each model generated in each fold.

Using validation set for testing the generalization power of model

1 Answers1