Can we create saliency maps from image segmentation tasks?

Question

I also know the fact that saliency map is also a form of image segmentation task. But it has been used very widely for interpretable deep learning ( Read GradCam etc ) . I also came across this paper (http://img.cs.uec.ac.jp/pub/conf16/161011shimok_0.pdf) which talks about Class Saliency Maps - something that rings a bell when it comes to Image Segmentation. Please tell if this concept exists for Image Segmentation or I need to read more on this subject.

Answer 1

Class saliency maps as described in Deep Inside Convolutional Networks: VisualisingImage Classification Models and Saliency Maps explain that such a map describes per pixel how much changing such a pixel will influence a prediction. Hence I see no reason why this could not be applied to image segmentation tasks.

The resulting images from the segmentation task and saliency map have to be interpreted differently however. In an image segmentation task the output is a per pixel prediction of whether or not a pixel belongs to a class, sometimes in the form of a certainty score.

A class saliency map describes per pixel how much changing that pixel would change the score of the classifier. Or quote from above paper: "which pixels need to be changed the least to affect the class score the most"

Edit: Added example. Say that a pixel gets a score of 99% for being of the class "Dog", we can be rather certain that this pixel actually is part of a dog. The salience map can show a low score for this same pixel. This means that changing this pixel slightly would not influence the prediction of that pixel belonging to the class "Dog". In my experience so far, both the per pixel class probability map and the salience map show somewhat similar patterns, but this does not mean they are to be interpreted equal.

A piece of code I came across that can be applied to pytorch models (from Nikhil Kasukurthi, not mine) can be found on github.