0

I have a multilabel data, with 8 labels for each instance.

The shape of validation data is (1000,8), when i use the data, keras fit_generator show me good result.

Epoch 1/5
 - 445s - loss: 0.3608 - acc: 0.8522 - val_loss: 0.3528 - val_acc: 0.8538
Epoch 2/5
 - 382s - loss: 0.3579 - acc: 0.8528 - val_loss: 0.3528 - val_acc: 0.8538
Epoch 3/5
 - 377s - loss: 0.3555 - acc: 0.8525 - val_loss: 0.3516 - val_acc: 0.8538
Epoch 4/5
 - 375s - loss: 0.3550 - acc: 0.8526 - val_loss: 0.3509 - val_acc: 0.8538
Epoch 5/5
 - 373s - loss: 0.3515 - acc: 0.8528 - val_loss: 0.3454 - val_acc: 0.8538
<keras.callbacks.History at 0x7fd7d2fca2b0>

But when i check the accuracy usig sklearn using following code
y_pred = model.predict_generator(valid_gen,steps=step_size_val)
I got this error ValueError: Classification metrics can't handle a mix of multilabel-indicator and continuous-multioutput targets

So i do 

y_pred[y_pred>=0.5] = 1
y_pred[y_pred<0.5] = 0
y_pred=y_pred.astype(int)

But now i got zero accuracy

Vivek Mehta
  • 2,612
  • 2
  • 18
  • 30
Talha Anwar
  • 2,699
  • 4
  • 23
  • 62
  • It is probably expected, for multi-label classification [sklearn.metrics.accuracy_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html) measures subset accuracy: the set of labels predicted for a sample must exactly match the corresponding set of labels in y_true. – Vivek Mehta Feb 01 '20 at 15:00
  • it mean for 100% accuracy (theoratically); if y_true is `[1,0,0,0,1,0,0,0]` then y_pred should be `[1,0,0,0,1,0,0,0]`. So how keras is giving good accuracy and what is the solution – Talha Anwar Feb 01 '20 at 15:06
  • yes, if one of the label does not match then for this particular example score is zero. – Vivek Mehta Feb 01 '20 at 15:08
  • Keras is calculating like this: if `y_true= [1,0,0,0,1,0,0,0]` and `y_pred = [0,0,0,0,1,0,0,0]` then accuracy is 0.875 (7 out of 8 is correct). – Vivek Mehta Feb 01 '20 at 15:10
  • though other matches, let suppose y_true is `[1,0,0,0,1,0,0,0] ` and y_pred is `[1,0,0,0,0,0,0,0]` , then result should be 7\8 or zero? – Talha Anwar Feb 01 '20 at 15:10
  • What this result should be that depends on your use case, You have both the metrics it's up to you, how you want to measure your model's performance. – Vivek Mehta Feb 01 '20 at 15:13
  • I just checked a [blog](https://blog.mimacom.com/text-classification/) , they use `loss='binary_crossentropy',` and `metrics=['categorical_accuracy']` for multilabel. And i am using simple `["accuracy"]` as metrics as in this [blog](https://www.pyimagesearch.com/2018/05/07/multi-label-classification-with-keras/) , so which one should i opt – Talha Anwar Feb 01 '20 at 15:14

0 Answers0