H2OResponseError in Grid Search Get Grid Sorting

Question

When I run:

data_h = h2o.H2OFrame(data)

### Edit: added asfactor() below to change integer target array.
data_h["BPA"] = data_h["BPA"].asfactor()

train, valid = data_h.split_frame(ratios=[.7], seed = 1234)
features = ["bq_packaging_consumepkg", "bq_packaging_microwave_v3", "bq_packaging_plasticbottle_v2", 
              "bq_packaging_hotdrink_v3", "bq_packaging_microwsaran_v3","bq_food_cannedfoods_v2"]
target = "BPA"

# Hyperparameter tuning
params = {"ntrees": [50, 100, 200, 300, 400, 500, 600],
          "max_depth": [10, 30, 50, 70, 90, 110],
          "min_rows": [1,5,10,15,20,25]}

criteria = {"strategy": "RandomDiscrete", 
            "stopping_rounds": 10,
            "stopping_tolerance": 0.00001,
            "stopping_metric": "misclassification"}

# Grid search and Training
grid_search = H2OGridSearch(model= rf_h, hyper_params= params, 
search_criteria = criteria)
grid_search.train(x = features, y = target, training_frame=train, 
validation_frame = valid)

# Sorting the grid
sorted_grid = grid_search.get_grid(sort_by='auc', decreasing = True)

Calling grid_search.get_grid(sort_by = 'auc', decreasing = True) produces the following error:

  H2OResponseError: Server error water.exceptions.H2OIllegalArgumentException:
  Error: Invalid argument for sort_by specified. Must be one of: [mae, residual_deviance, r2, mean_residual_deviance, rmsle, rmse, mse]
  Request: GET /99/Grids/Grid_DRF_py_29_sid_95b5_model_python_1533334963198_8
    params: {'sort_by': 'auc', 'decreasing': 'True'}

Looking at the example in the documentation for the grid search I believe that I am using the method correctly.

Edit: Added changing target array to be a factor array from an integer array.

score 2 · Answer 1 · answered Aug 07 '18 at 21:40

2

Error: Invalid argument for sort_by specified. Must be one of: [mae, residual_deviance, r2, mean_residual_deviance, rmsle, rmse, mse]

The problem is that "auc" is not a valid metric for your problem. It looks like you have trained a regression model instead of a binary classification model, that's why AUC is not allowed. The list of metrics in the error message is the list of allowed metrics for a regression problem.

If your response column is 0's and 1's and you did not convert it to a factor, then it's going to train a regression model instead of a binary classfication model. If this is the case, and you want a binary classification model instead, then all you need to do is first convert the response to a factor:

train[target] = train[target].asfactor()

answered Aug 07 '18 at 21:40

Erin LeDell

8,704
1
19
35

thanks for pointing out that situation, I indeed had integers representing the targets for classification. I tried passing in the `asfactor()` code before training, but the same error persists. I have included more code in my question. – Roy Z Aug 08 '18 at 16:51
Can you add the output of: `train.type(target)` and `train[target].table()`? – Erin LeDell Aug 08 '18 at 17:33
`train.type(target)` outputs `enum` and `train[target].table()` outputs a [table with each factor level](https://imgur.com/a/CmGIPM6) and its counts (BPA is my target). – Roy Z Aug 08 '18 at 17:38
Please confirm that you have only two factor levels in your response. If so, next try: `grid_search[0].auc()` and see if it outputs anything. And then can you re-start your H2O cluster (or delete the grid) and try running your code again? – Erin LeDell Aug 08 '18 at 18:56
I have restarted the cluster and retrained the model and the error still persists. I ran `grid_search[0].auc()` and I got a `KeyError: 'AUC'` – Roy Z Aug 08 '18 at 21:10
@RoyZ I can't reproduce this, so unless you can provide a reproducible example, I don't think I can help. Sorry! – Erin LeDell Aug 20 '18 at 21:28
no worries, the problem was resolved, like Lauren was saying, it was a multiple classification task so AUC would not work. – Roy Z Aug 20 '18 at 23:58

score 2 · Accepted Answer · answered Aug 10 '18 at 15:23

2

This particular question is asking how to get the AUC for a multiclass classification problem (i.e. the target has more than two factor levels - see the posted image in the comments of the original question). H2O does not calculate the auc for individual categories, and therefore will return an error if you try to use its binary-classification metric auc().

To see what metrics are available for multiclass classification problems please see the documentation

Options include, for example: logloss() and mean_per_class_error()

answered Aug 10 '18 at 15:23

Lauren

5,640
1
13
19

so I should be running something like `grid_search.get_grid(sort_by='logloss, decreasing = True)`? – Roy Z Aug 10 '18 at 15:33
yes now that your target is categorical H2O should accept logloss as a sort_by metric. – Lauren Aug 10 '18 at 15:39
Sorry for the delayed response. This worked and your answer has been marked correct. Thanks! – Roy Z Aug 16 '18 at 18:51

H2OResponseError in Grid Search Get Grid Sorting

2 Answers2