Incremental training of random forest model using python sklearn

Question

I am using the below code to save a random forest model. I am using cPickle to save the trained model. As I see new data, can I train the model incrementally. Currently, the train set has about 2 years data. Is there a way to train on another 2 years and (kind of) append it to the existing saved model.

rf =  RandomForestRegressor(n_estimators=100)
print ("Trying to fit the Random Forest model --> ")
if os.path.exists('rf.pkl'):
    print ("Trained model already pickled -- >")
    with open('rf.pkl', 'rb') as f:
        rf = cPickle.load(f)
else:
    df_x_train = x_train[col_feature]
    rf.fit(df_x_train,y_train)
    print ("Training for the model done ")
    with open('rf.pkl', 'wb') as f:
        cPickle.dump(rf, f)
df_x_test = x_test[col_feature]
pred = rf.predict(df_x_test)

EDIT 1: I don't have the compute capacity to train the model on 4 years of data all at once.

score 13 · Accepted Answer · edited Aug 13 '19 at 18:59

What you're talking about, updating a model with additional data incrementally, is discussed in the sklearn User Guide:

Although not all algorithms can learn incrementally (i.e. without seeing all the instances at once), all estimators implementing the partial_fit API are candidates. Actually, the ability to learn incrementally from a mini-batch of instances (sometimes called “online learning”) is key to out-of-core learning as it guarantees that at any given time there will be only a small amount of instances in the main memory.

They include a list of classifiers and regressors implementing partial_fit(), but RandomForest is not among them. You can also confirm RFRegressor does not implement partial fit on the documentation page for RandomForestRegressor.

Some possible ways forward:

Use a regressor which does implement partial_fit(), such as SGDRegressor
Check your RandomForest model's feature_importances_ attribute, then retrain your model on 3 or 4 years of data after dropping unimportant features
Train your model on only the most recent two years of data, if you can only use two years
Train your model on a random subset drawn from all four years of data.
Change the tree_depth parameter to constrain how complicated your model can get. This saves computation time and so may allow you to use all your data. It can also prevent overfitting. Use Cross-Validation to select the best tree-depth hyperparameter for your problem
Set your RF model's param n_jobs=-1 if you haven't already,to use multiple cores/processors on your machine.
Use a faster ensemble-tree-based algorithm, such as xgboost
Run your model-fitting code on a large machine in the cloud, such as AWS or dominodatalab

Would SGD be a good alternative to random forest ? I am not sure how the new model would perform. SGD has a partial_fit API, so I can use it for Online learning — ForeverLearner, May 19 '17 at 05:25
Have you tried training an SGD classifier and comparing the performance to the RF model's performance? That's the only way to answer that question — Max Power, May 19 '17 at 14:55
Thanks. I am trying to find the important features and see if I can fit all 4 years into available RAM. — ForeverLearner, May 21 '17 at 10:02
The parameter warm_start allows adding new estimators to the model based on the new coming data. This can be used for incremental learning. — André Gomes, Aug 24 '18 at 16:01

M. Gopal · Answer 2 · 2018-05-10T06:40:02.977

You can set the 'warm_start' parameter to True in the model. This will ensure the retention of learning with previous learn using fit call.

Same model learning incrementally two times (train_X[:1], train_X[1:2]) after setting ' warm_start '

forest_model = RandomForestRegressor(warm_start=True)
forest_model.fit(train_X[:1],train_y[:1])
pred_y = forest_model.predict(val_X[:1])
mae = mean_absolute_error(pred_y,val_y[:1])
print("mae      :",mae)
print('pred_y :',pred_y)
forest_model.fit(train_X[1:2],train_y[1:2])
pred_y = forest_model.predict(val_X[1:2])
mae = mean_absolute_error(pred_y,val_y[1:2])
print("mae      :",mae)
print('pred_y :',pred_y)

mae : 1290000.0 pred_y : [ 1630000.] mae : 925000.0 pred_y : [ 1630000.]

Model only with the last learnt values ( train_X[1:2] )

forest_model = RandomForestRegressor()
forest_model.fit(train_X[1:2],train_y[1:2])
pred_y = forest_model.predict(val_X[1:2])
mae = mean_absolute_error(pred_y,val_y[1:2])
print("mae      :",mae)
print('pred_y :',pred_y)

mae : 515000.0 pred_y : [ 1220000.]

http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html

warm_state, just add more estimators. That means using only the new data, it will train a new estimator. And later decision is collectively made i.e. based upon the old+new estimators result. It wont trained a existing model, just merely add new estimators. — vipin bansal, Jun 07 '19 at 12:18
I tried this method, but I got `ValurError : Class label 21 not present`. How to resolve such type of issue. — Aditya Sharma, Jun 23 '20 at 12:48

Incremental training of random forest model using python sklearn

2 Answers2

Linked