0

I am using H2o's Auto ML package and would like to know if it is possible to get a single AUC, Confusion Matrix and ROC curve for all the methods combined. For instance I have AUC values for the individual models GLM, Stacked Ensemble, deep learning etc. Can you get these three values for all the methods combined? The goal is to be able to compare the Auto ML package to other similar packages.

Kevin
  • 1
  • 1
  • What do you mean by "all the methods combined"? Can you edit your question be more specific? – Erin LeDell Jun 21 '18 at 20:26
  • After running auto ml the models that had the highest AUC were GLM grid, stackedEnsemble Allmodels, stackedEnsemble bestoffamily, deeplearning, DRF and XRT. I would like to get an AUC, ROC and confusion matrix for all these models together. Can I average them with equal weights and then make the AUC, ROC and confusion matrix? I want to compare the AutoML process to individual RF, SVM only models. – Kevin Jun 22 '18 at 03:08
  • 1
    Also I couldn't find a way to get sensitivity, specificity, recall, precision, and Matthews correlation coefficient values. Is this possible? – Kevin Jun 22 '18 at 03:17
  • you can get these values as metric attributes for each of the model, see the docs here: http://docs.h2o.ai/h2o/latest-stable/h2o-py/docs/metrics.html?highlight=sensitivity#h2o.model.metrics_base.H2OBinomialModelMetrics.sensitivity – Lauren Jun 22 '18 at 21:11
  • Are you saying you want to draw, say, 3 lines on the AUC chart to compare models? Or are you saying you want to make an ensemble of the autoML models, and see how that does? (It might be helpful to show the output of the "similar packages" you want to compare automl to.) – Darren Cook Jun 26 '18 at 09:30
  • Yes I plan on drawing the AUC lines for several different models to compare performance. What I am unsure about is how to compare the Auto ML package to the other ones I have. Can I do an equal weight average of the top five models from the package and use them as an indicator of performance? I want to show that the Auto ML package is better but because it returns multiple methods and their individual AUC statistics I am unsure how to come up with a performance metric. – Kevin Jun 27 '18 at 18:32
  • Using the references above I was able to save the top model and get the AUC curve, model statistics and a Predicted vs measured plot. Thank you for the help! – Kevin Aug 02 '18 at 16:20

0 Answers0