3

I'm using a logistic regression model in sklearn and I am interested in retrieving the log likelihood for such a model, so to perform an ordinary likelihood ratio test as suggested here.

The model is using the log loss as scoring rule. In the documentation, the log loss is defined "as the negative log-likelihood of the true labels given a probabilistic classifier’s predictions". However, the value is always positive, whereas the log likelihood should be negative. As an example:

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss

lr = LogisticRegression()
lr.fit(X_train, y_train)
y_prob = lr.predict_proba(X_test)
log_loss(y_test, y_prob)    # 0.66738

I do not see any method in the documentation for the model, is there any other possibility that I am currently not aware of?

desertnaut
  • 57,590
  • 26
  • 140
  • 166
Mattia Paterna
  • 1,268
  • 3
  • 15
  • 31

2 Answers2

9

Read closely; the log loss is the negative log-likelihood. Since log-likelihood is indeed as you say negative, its negative will be a positive number.

Let's see an example with dummy data:

from sklearn.metrics import log_loss
import numpy as np

y_true = np.array([0, 1, 1])
y_pred = np.array([0.1, 0.2, 0.9])

log_loss(y_true, y_pred)
# 0.60671964791658428

Now, let's compute manually the log-likelihood elements (i.e. one value per label-prediction pair), using the formula given in the scikit-learn docs you have linked to without the minus sign:

log_likelihood_elements = y_true*np.log(y_pred) + (1-y_true)*np.log(1-y_pred)
log_likelihood_elements
# array([-0.10536052, -1.60943791, -0.10536052])

Now, given the log-likelihood elements (which are indeed negative), the log loss is the negative of their sum, divided by the number of samples:

-np.sum(log_likelihood_elements)/len(y_true)
# 0.60671964791658428

log_loss(y_true, y_pred) == -np.sum(log_likelihood_elements)/len(y_true)
# True
desertnaut
  • 57,590
  • 26
  • 140
  • 166
  • Thank you so much! I am pretty stupid today. That means, the lower the better when using the log loss. Or I might change sign and treat it as log likelihood - right? – Mattia Paterna Jan 10 '18 at 11:30
  • 4
    @MattiaPaterna Every quantity described as "loss", implies "the lower the better" – desertnaut Jan 10 '18 at 11:39
1

To get the log likelihood you can calculate:

-log_loss(y_true, y_pred)*len(y_true)
moruobai
  • 11
  • 1