2

I am trying to train a pre-trained VGG16 model from Keras for a multi-class multi-label classification task. The images are from Chest X-Ray 8 dataset from NIH. The dataset has 14 labels (14 diseases) plus a "no finding" label.

I understand that for independent labels, like the 14 diseases, I should use sigmoid activation + binary_crossentropy loss function; and for dependent labels, I should use softmax + categorical_crossentropy.

However, out of my 15 labels in total, 14 of them are independent, but the one "no finding" is technically dependent with the rest 14 --> the probability of "no finding" and having disease(s) should add up to 1, but the probability of having what disease(s) should be given independently. So what loss should I use?

Besides, my output is a list of floats(probabilities), each column is a label.

y_true:
[[0. 0. 0. ... 0. 0. 1.]
 [0. 0. 0. ... 0. 0. 1.]
 [0. 0. 1. ... 0. 0. 0.]
 ...
 [0. 0. 0. ... 0. 0. 1.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 1.]]

y_predict:
[[0.1749 0.0673 0.1046 ... 0.     0.     0.112 ]
 [0.     0.1067 0.2804 ... 0.     0.     0.722 ]
 [0.     0.     0.0686 ... 0.     0.     0.5373]
 ...
 [0.0571 0.0679 0.0815 ... 0.     0.     0.532 ]
 [0.0723 0.0555 0.2373 ... 0.     0.     0.4263]
 [0.0506 0.1305 0.4399 ... 0.     0.     0.2792]]

Such result makes it impossible to use classification_report() function to evaluate my model. I am thinking about getting a threshold to transfer it to binary, but it will be more human-modification instead of CNN prediction, as I have to select a threshold. So I am unsure whether I should do some hard-code stuff or are there any other already-exist methods to deal with this situation?

I am quite new to CNN and classification, so if anyone can guide me or give me any hint I will appreciate it very much. Thank you!

Main body code as below:

vgg16_model = VGG16()
last_layer = vgg16_model.get_layer('fc2').output

#I am treating them all as independent labels
out = Dense(15, activation='sigmoid', name='output_layer')(last_layer)
custom_vgg16_model = Model(inputs=vgg16_model.input, outputs=out)

for layer in custom_vgg16_model.layers[:-1]:
layer.trainable = False

custom_vgg16_model.compile(Adam(learning_rate=0.00001), 
                           loss = "binary_crossentropy", 
                           metrics = ['accuracy'])   # metrics=accuracy gives me very good result, 
                                                     # but I suppose it is due to the large amount 
                                                     # of 0 label(not-this-disease prediction),
                                                     # therefore I am thinking to change it to 
                                                     # recall and precision as metrics. If you have
                                                     # any suggestion on this I'd also like to hear!
Jasminy
  • 109
  • 7

1 Answers1

1

Some update about my project, and I have actually managed to solve most of the problems mentioned in this question.

Firstly, as this is a multi-class multi-label classification question, I decided to use ROC-AUC score instead of precision or recall as the evaluation metrics. The advantage of it is that there is no threshold value involved -- AUC is the a bit like an average of performance under a range of thresholds. And it only looks at the positive prediction, so it reduces the effect of the majority of 0s in the dataset. This gives a more accurate prediction of the model's performance in my case.

For the output class, I decided to use 14 classes instead of 15 -- if all labels are 0 then it means "no finding". Then I can happily use sigmoid activation in my output layer. Despite it, I use focal loss instead of binary cross entropy as my dataset is highly imbalanced.

I still face problem as my ROC is not good (very close to y=x and sometimes below y=x). But I hope my progress can give anyone who find this some inspiration.

Jasminy
  • 109
  • 7