0

I have trained three different models separately in caffe, and I can get the probability of belonging to each class for semantic segmentation. I want to get an output based on the 3 probabilities that I am getting (for example, the argmax of three probabilities). This can be done by inferring through net model and deploy.prototxt files. And then based on the final soft output, the hard output shows the final segmentation. My questions are:

  1. How to get ensemble output of these networks?
  2. How to do end-to-end training of ensemble of three networks? Is there any resources to get help?
  3. How to get final segmentation based on the final probability (e.g., argmax of three probabilities), which is soft output?

My question may sound very basic question, and my apologies for that. I am still trying to learn step by step. I really appreciate your help.

S.EB
  • 1,966
  • 4
  • 29
  • 54

1 Answers1

1

There are two ways (at least that I know of) that you could do to solve (1):

  1. One is to use pycaffe interface, instantiate the three networks, forward an input image through each of them, fetch the output and perform any operation you desire to combine all three probabilites. This is specially useful if you intend to combine them using a more complex logic.

  2. The alternative (way less elegant) is to use caffe test and process all your inputs separately through each network saving the probabilities into files. Then combine the probabilities from the files later.

Regarding your second question, I have never trained more than two weight-sharing CNNs (siamese networks). From what I understood, your networks don't share weights, only the architecture. If you want to train all three end-to-end please take a look at this tutorial made for siamese networks. The authors define in their prototxt both paths/branches, connect each branch's layers to the input Data layer and, at the end, with a loss layer.

In your case you would define the three branches (one for each of your networks), connect with input data layers (check if each branch processes the same input or different inputs, for example, the same image pre-processed differently) and unite them with a loss, similarly to the tutorial.

Now, for the last question, it seems Caffe has a ArgMax layer that may be what you are looking for. If you are familiar with python, you could also use a python layer that allows you to define with great flexibility how to combine the output probabilities.

rafaspadilha
  • 629
  • 6
  • 20
  • Thanks a lot for your help, I have performed the offline inference for three different inputs, which is a 3D data and each input corresponds to a specific plane. I take one plane fixed (xy), and then I do stacking probabilities on the other two (yz,xz) to bring their shape as the shape of xy. I have got the `probabilities` based on `Softmax` layer in `deploy.prototxt`. Could I ask how can I use `ArgMax Layer` in this case. since the shapes and inputs are not same. Could you please explain more on this? Thanks a lot – S.EB Feb 02 '18 at 13:09
  • In this case, I cannot use `ArgMax` in `deploy.prototxt`, right? I do not know how to get the final class label by getting the maximum value of three probabilities? Thanks for your help – S.EB Feb 02 '18 at 13:14
  • You will use the `ArgMax layer` after the `Softmax` of each CNN. Let's say that, after `Softmax`, you have a tensor of dimensions 3x1x1 (1x1 because the softmax outputs a single value, *i.e.* the soft probability; and x3 because it is 3 CNNs/soft probabilities), the `ArgMax` can select the index (in case you need to know which soft prob was higher) and the value of the maxed probability. – rafaspadilha Feb 05 '18 at 11:27