1

The exported pipeline of TPOT stating that the Average CV score on the training set was: -128.90187963562252 (neg_MAE). However, refitting the pipeline with the same exact training set yields way smaller MAE around (35). Moreover predicting unseen test set would yield an MAE around (140) which is in line with what the exported pipeline stating.

I am a bit confused and wondering how to reproduce the error score on the training set.

The pipeline seems to be overfitting right??

cv = RepeatedKFold(n_splits=4, n_repeats=1, random_state=1)
model = TPOTRegressor(generations=10, population_size=25, offspring_size=None, mutation_rate=0.9,
                      crossover_rate=0.1, scoring='neg_mean_absolute_error', cv=cv, 
                      subsample=0.75,n_jobs=-1, max_time_mins=None, 
                      max_eval_time_mins=5,random_state=42,config_dict=None, template=None, 
                      warm_start=False, memory=None, 
                      use_dask=False,periodic_checkpoint_folder=None, early_stop=3, verbosity=2,
                      disable_update_check=False, log_file=None)

model.fit(train_df[x], train_df[y])

# The Exported model
# Average CV score on the training set was: -128.90187963562252

exported_pipeline = make_pipeline(StackingEstimator(estimator=LassoLarsCV(normalize=True)),
                                  StackingEstimator(estimator=ExtraTreesRegressor(bootstrap=True, 
                                   max_features=0.4, min_samples_leaf=1,
                                   min_samples_spli`enter code here`t=7, n_estimators=100)),
                                   PolynomialFeatures(degree=2, include_bias=False, 
                                   interaction_only=False),
                                   ExtraTreesRegressor(bootstrap=True, 
                                   max_features=0.15000000000000002, min_samples_leaf=9, 
                                   min_samples_split=7,n_estimators=100))

# Fix random state for all the steps in exported pipeline

set_param_recursive(exported_pipeline.steps, 'random_state', 42)
exported_pipeline.fit(training_features, training_target) 
results = exported_pipeline.predict(testing_features)

Thanks in advance

Ghali Yakoub
  • 11
  • 3
  • I think there's an error in the last line of your code there. That doesn't look like valid code. Can you please review it and fix it if necessary? – Randy Olson Apr 23 '21 at 15:15
  • Hi Randy, the code is fine, it is just when I post it here. it got messed up. I will fix it. Nevertheless, regarding the different score I got after refitting, it seems that stacking the estimators produces almost no error on the training dataset after refitting. I am still trying to figure out exactly what is going on – Ghali Yakoub Apr 23 '21 at 19:55
  • Great. Can you also please share the code that you use to find the -35 negative MAE? If I understand your post right right, that's from fitting `exported_pipeline` on the training data then also scoring `exported_pipeline` on the training data, right? – Randy Olson Apr 24 '21 at 20:11
  • Hi Andy, that is correct. – Ghali Yakoub Apr 25 '21 at 16:38
  • I have been working on it, and I noticed that stcaking the estimators produced overfitting however I am still not sure why the exported file stating that the MAE is more or less in the same range of error on an unseen dataset. – Ghali Yakoub Apr 25 '21 at 16:40
  • below is the new code: I just changed the scoring to be the Huber and disabled the stacking – Ghali Yakoub Apr 25 '21 at 16:41
  • from scipy.special import huber from sklearn.metrics import make_scorer def my_custom_huber(y_true, y_pred): return huber(50,(y_true-y_pred)).mean() my_custom_scorer = make_scorer(my_custom_huber, greater_is_better=False) – Ghali Yakoub Apr 25 '21 at 16:43
  • model = TPOTRegressor(generations=5, population_size=50, offspring_size=10, mutation_rate=0.9, crossover_rate=0.1, scoring=my_custom_scorer, cv=3, subsample=1, n_jobs=-1, max_time_mins=None, max_eval_time_mins=5,random_state=42, config_dict=None, template='Selector-Transformer-Regressor', warm_start=False, memory='auto', use_dask=False, periodic_checkpoint_folder=None, early_stop=2, verbosity=3, disable_update_check=False, log_file=None ) – Ghali Yakoub Apr 25 '21 at 16:43
  • Here, I was able to reproduce the results and have a more or less perfect fitted model. Btw, I am working with data from a wind farm and unfortunately I can't share it – Ghali Yakoub Apr 25 '21 at 16:45

0 Answers0