0

I tried SageMaker's AutoPilot to solve a binary classification problem and I found it is using f1 as the evaluation metric. But when I tried to write some code without tuning like this:

xgb.set_hyperparameters(max_depth=5,
                        eta=0.2,
                        gamma=4,
                        min_child_weight=6,
                        subsample=0.8,
                        objective='binary:logistic',
                        eval_metric='f1',
                        num_round=100)

This generates the following error:

[2021-10-17:00:02:19:ERROR] Customer Error: Metric 'f1' is not supported. Parameter 'eval_metric' should be one of these options:'rmse', 'mae', 'logloss', 'error', 'merror', 'mlogloss', 'auc', 'ndcg', 'map', 'poisson-nloglik', 'gamma-nloglik', 'gamma-deviance', 'tweedie-nloglik'.

Since the autopilot was able to compute F1, I feel like it is supported in the hyperparameter setting in some fashion? Am I misunderstanding?

Any help is going to be appreciated.

Flavia Giammarino
  • 7,987
  • 11
  • 30
  • 40
kee
  • 10,969
  • 24
  • 107
  • 168
  • 1
    use validation:f1 with a validation channel. https://docs.aws.amazon.com/sagemaker/latest/dg/xgboost-tuning.html – Neil McGuigan Oct 18 '21 at 14:26
  • Thanks. I actually read the same article and tried eval_metric="validation:f1" but I got the same error. When you say "a validation channel", what does that mean? – kee Oct 18 '21 at 18:39
  • 1
    I believe you can set eval_metric="error" in a single training job, but can set validation:f1 in a hyperparameter tuning job – Neil McGuigan Oct 18 '21 at 21:26

1 Answers1

0

You can define the metrics that you want to send to CloudWatch by specifying a list of metric names and regular expressions as the metric_definitions argument when you initialize an Estimator object. See it here: Documentation

import sagemaker
from sagemaker.estimator import Estimator

estimator = Estimator(
    image_uri="your-own-image-uri",
    role=sagemaker.get_execution_role(), 
    sagemaker_session=sagemaker.Session(),
    instance_count=1,
    instance_type='ml.c4.xlarge',
    metric_definitions=[
       {'Name': 'train:error', 'Regex': 'Train_error=(.*?);'},
       {'Name': 'validation:error', 'Regex': 'Valid_error=(.*?);'}
    ]
)
Yane
  • 1