How to write a custom loss function in LGBM?

Question

I have a binary cross-entropy implementation in Keras. I would like to implement the same one in LGBM as a custom loss. Now I understand LGBM of course has 'binary' objective built-in but I would like to implement this one custom-made on my own as a starter for some future enhancements.

Here is the code,

def custom_binary_loss(y_true, y_pred): 
    """
    Keras version of binary cross-entropy (works like charm!)
    """
    # https://github.com/tensorflow/tensorflow/blob/v2.3.1/tensorflow/python/keras/backend.py#L4826
    y_pred = K.clip(y_pred, K.epsilon(), 1 - K.epsilon())
    
    term_0 = (1 - y_true) * K.log(1 - y_pred + K.epsilon())  # Cancels out when target is 1 
    term_1 = y_true * K.log(y_pred + K.epsilon()) # Cancels out when target is 0

    return -K.mean(term_0 + term_1, axis=1)

# --------------------
def custom_binary_loss_lgbm(y_pred, train_data):
    """
    LGBM version of binary cross-entropy
    """
    y_pred = 1.0 / (1.0 + np.exp(-y_pred))

    y_true = train_data.get_label()
    y_true = np.expand_dims(y_true, axis=1)
    y_pred = np.expand_dims(y_pred, axis=1)
    
    epsilon_ = 1e-7
    y_pred = np.clip(y_pred, epsilon_, 1 - epsilon_)

    term_0 = (1 - y_true) * np.log(1 - y_pred + epsilon_)   # Cancels out when target is 1 
    term_1 = y_true * np.log(y_pred + epsilon_)  # Cancels out when target is 0

    grad = -np.mean(term_0 + term_1, axis=1)
    hess = np.ones(grad.shape)
    return grad, hess

But using the above my LGBM model only predicts zeros. Now my dataset is balanced and everything looks cool so what's the error here?

params = {
    'objective': 'binary',
    'num_iterations': 100,
    'seed': 21
}
ds_train = lgb.Dataset(df_train[predictors], y, free_raw_data=False)
reg_lgbm = lgb.train(params=params, train_set=ds_train, fobj=custom_binary_loss_lgbm)

I also tried with a different hessian hess = (y_pred * (1. - y_pred)).flatten(). Although I don't know what hessian really means it didn't work either!

list(map(lambda x: 1.0 / (1.0 + np.exp(-x)), reg_lgbm.predict(df_train[predictors])))

[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,  .............]

score 1 · Answer 1 · answered Jun 02 '21 at 11:45

Try setting the metric parameter to the string "None" in params, like this:

params = {
    'objective': 'binary',
    'metric': 'None',
    'num_iterations': 100,
    'seed': 21
}

Otherwise, according to the documentation, the algorithm would choose a default evaluation method for objective set to 'binary'

How to write a custom loss function in LGBM?

1 Answers1