4

I would like to train a network with multiple output layers.

in->hidden->out 1
          ->out 2

Is this possible? If so how do I setup the datasets and trainer to accomplish training.

Firestrand
  • 1,305
  • 10
  • 19

3 Answers3

1

As you are looking into splitting your output in order to have several SoftMax regions, you can use PartialSoftmaxLayer provided by PyBrain.

Note that it is limited to slices of the same length, but its code can inspire you if you require a custom output layer:

https://github.com/pybrain/pybrain/blob/master/pybrain/structure/modules/softmax.py

jjmontes
  • 24,679
  • 4
  • 39
  • 51
0

No. You can have multiple hidden layers, like this

in -> hidden 1 -> hidden 2 -> out

Alternatively, you can have multiple output neurons (in a single output layer).

Technically, you can set up any arrangement of neurons and layers, connect them however you like, and call them whatever you want, but the above is the general way of doing it.

andrelucas
  • 678
  • 3
  • 10
  • My question was about output layers not hidden layers and my problem is training them. It doesn't appear that pybrain will allow for training multiple output layers. – Firestrand Mar 09 '14 at 05:23
  • The problem is, how is the training error calculated? The error is the difference between the ideal and actual output. If you have 2 output layers, which one do you use? If you use both, its essentially the same as combining the neurons from your two output layers into one larger layer, as @lmjohns3 mentioned. – andrelucas Mar 09 '14 at 12:26
0

It would be more work for you as the programmer, but if you want to have two different outputs, you can always concatenate your outputs into one vector and use that as the output for the network.

in --> hidden --> concatenate([out1, out2])

A possibly significant drawback of this approach is that if the two outputs are of different scales, then concatenation will distort the error metric you use to train the network.

However, if you were able to use two separate outputs, then you'd still need to solve this problem, likely by somehow weighting the two error metrics that you use.

Potential solutions to this problem could include defining a custom error metric (e.g., by using a variant of weighted squared error or weighted cross-entropy) and/or standardizing the two output datasets so that they exist in a common scale.

lmjohns3
  • 7,422
  • 5
  • 36
  • 56
  • 1
    My current need is for 2 output softmax layers so I can't concatenate them. I agree however in other cases I could simply combine my outputs into one layer. As for scales there are of course numerous scaling methods to ensure that's not a problem. – Firestrand Mar 09 '14 at 05:25