GridSearchCV is very slow to estimate my model

Question

I am using GridSearchCV on an MLP Classifier, this is my code...

normalized_features.shape # (50000,784)
len(labels)               #  50000
X_train, X_test, Y_train, Y_test = train_test_split(normalized_features, labels, test_size=0.2)
mlp = MLPClassifier(max_iter=100)
parameter_space = {
    'hidden_layer_sizes': [(50,50,50), (50,100,50), (100,)],
    'activation': ['tanh', 'relu'],
    'solver': ['sgd', 'adam'],
    'alpha': [0.0001, 0.05],
    'learning_rate': ['constant','adaptive'],
}

This is the stage where I got struck, It's been more than two hours and still it keeps on loading and throws warnings

clf = GridSearchCV(mlp, parameter_space, n_jobs=-1, cv=10)
clf.fit(X_train, Y_train)

Warnings:

/usr/local/lib/python3.6/dist-packages/joblib/externals/loky/process_executor.py:706: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak. "timeout or by a memory leak.", UserWarning

Can anyone please help me out with this and let me know where did I go wrong! Thank you in advance.

https://github.com/ray-project/tune-sklearn - this package speeds up large hyperparameter tuning processes — noobProgrammer, Jul 09 '20 at 04:27

score 1 · Answer 1 · answered Feb 29 '20 at 06:05

You are training (train and validation) on 50000 samples of 784 features over the parameter space of 3 x 2 x 2 x 2 x 3 = 72 with CV of 10, which mean you are training 10 model each 72 times. Run it once with one set of parameters and and you can roughly extrapotate how much time it will take for your setup. It will take time for sure.

score 1 · Answer 2 · answered Feb 29 '20 at 14:36

As @mujjiga mentioned Gridsearch will try 72 different parameter combination for your each fold and if you have 10 fold the total model number that will be trained is 720.

You may be want to use RandomizedSearch which will give you similar result with GridSearch method with less experiment. So you can decrease the time of your training.

You can find randomizedsearch implementation in scikit-learn.

Also you can read the comparision between gridsearch and randomizedsearch via this link with more detail.

GridSearchCV is very slow to estimate my model

2 Answers2