log_loss in sklearn: Multioutput target data is not supported with label binarization

Question

The following code

from sklearn import metrics
import numpy as np
y_true = np.array([[0.2,0.8,0],[0.9,0.05,0.05]])
y_predict = np.array([[0.5,0.5,0.0],[0.5,0.4,0.1]])
metrics.log_loss(y_true, y_predict)

produces the following error:

   ---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-32-24beeb19448b> in <module>()
----> 1 metrics.log_loss(y_true, y_predict)

~\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\sklearn\metrics\classification.py in log_loss(y_true, y_pred, eps, normalize, sample_weight, labels)
   1646         lb.fit(labels)
   1647     else:
-> 1648         lb.fit(y_true)
   1649 
   1650     if len(lb.classes_) == 1:

~\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\sklearn\preprocessing\label.py in fit(self, y)
    276         self.y_type_ = type_of_target(y)
    277         if 'multioutput' in self.y_type_:
--> 278             raise ValueError("Multioutput target data is not supported with "
    279                              "label binarization")
    280         if _num_samples(y) == 0:

ValueError: Multioutput target data is not supported with label binarization

I am curious why. I am trying to re-read definition of log loss and cannot find anything that would make computations incorrect.

Answer 1

The source code indicates that metrics.log_loss does not support probabilities in y_true. It only supports binary indicators of shape (n_samples, n_classes), for example [[0,0,1],[1,0,0]] or class labels of shape (n_samples,), for example [2, 0]. In the latter case the class labels will be one-hot encoded to look like the indicator matrix before calculating log loss.

In this block:

lb = LabelBinarizer()

if labels is not None:
    lb.fit(labels)
else:
    lb.fit(y_true)

You are reaching lb.fit(y_true), which will fail if y_true is not all 1 and/or 0. For example:

>>> import numpy as np
>>> from sklearn import preprocessing

>>> lb = preprocessing.LabelBinarizer()

>>> lb.fit(np.array([[0,1,0],[1,0,0]]))

LabelBinarizer(neg_label=0, pos_label=1, sparse_output=False)

>>> lb.fit(np.array([[0.2,0.8,0],[0.9,0.05,0.05]]))

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/imran/.pyenv/versions/anaconda3-4.4.0/lib/python3.6/site-packages/sklearn/preprocessing/label.py", line 278, in fit
    raise ValueError("Multioutput target data is not supported with "
ValueError: Multioutput target data is not supported with label binarization

I would define your own custom log loss function:

def logloss(y_true, y_pred, eps=1e-15):
    y_pred = np.clip(y_pred, eps, 1 - eps)
    return -(y_true * np.log(y_pred)).sum(axis=1).mean()

Here is the output on your data:

>>> logloss(y_true, y_predict)
0.738961717153653

Answer 2

No, I am not talking about binary classification.

The y_true and y_predict you showed above will not be considered as classification targets, unless otherwise specified as such.

First since they are probabilities so it can take any continuous values and hence it is detected as regression in scikit.

Second, each element inside y_pred or y_true is a list of probabilities. That is detected as multi-output. Hence the error of "Multioutput targets".

You need to supply the actual labels for log_loss and not probabilities for y_true (Ground truths). Why do you have probabilities in that, by the way?? Probabilities can be present for predicted data but why for actual data?

For that you need to first convert the probabilities of y_true into labels by considering the highest probability as the winner class.

This can be done by numpy.argmax by using the below code:

import numpy as np
y_true = np.argmax(y_true, axis=1)

print(y_true)
Output:-  [0, 1]
# We will not do this the above for y_predict, because probabilities are allowed in it.

# We will use labels param to declare that we have actually 3 classes, 
# as evident from your probabilities.
metrics.log_loss(y_true, y_predict, labels=[0,1,2])

Output:-  0.6931471805599458

As discussed with @Imran, here's an example having y_true with values other than 0 or 1.

• This example here uses log_loss for a 3-class classification in which y have values 0, 1 and 2 :- http://scikit-learn.org/stable/auto_examples/calibration/plot_calibration_multiclass.html#sphx-glr-auto-examples-calibration-plot-calibration-multiclass-py

Below example to simply check if other values are allowed or not:

y_true = np.array([0, 1, 2])
y_pred = np.array([[0.5,0.5,0.0],[0.5,0.4,0.1], [0.4,0.1,0.5]])
metrics.log_loss(y_true, y_pred)

Output:- 1.3040076684760489   (No error)