3

I am developing a program to study Neural Networks, by now I understand the differences (I guess) of dividing a dataset into 3 sets (training, validating & testing). My networks may be of just one output or multiple outputs, depending on the datasets and the problems. The learning algorithm is the back-propagation.

So, the problem basically is that I am getting confused with each error and the way to calculate it.

Which is the training error? If I want to use the MSE is the (desired - output)^2 ? But then, what happens if my network has 2 or more outputs, the training error is going to be the sum of all outputs?

Then, the validation error is just using the validation data set to calculate the output and compare the obtained results with the desired results, this will give me an error, is it computed the same way as in the training error? and with multiple outputs?

And finally, not totally clear, when should the validation run? Somewhere I read that it could be once every 5 epochs, but, is there any rule for this?

Thanks the time in advance!

gkapellmann
  • 313
  • 3
  • 19

1 Answers1

5

For multiple output neurons, to calculate the training error, in each epoch/iteration, you take each output value, get the difference to the target value for that neuron. Square it, do the same for the other output neurons, and then get the mean. So eg with two output neurons,

MSE = (|op1 - targ1|^2 + |op2 - targ2|^2 ) / 2

The training, validation and test errors are calculated the same way. The difference is when they are run and how they are used.
The full validation set is usually checked on every training epoch. Maybe to speed computation, you could run it every 5.
The result of the validation test/check is not used to update the weights, only to decide when to exit training. Its used to decide if the network has generalized on the data, and not overfitted.

Check the pseudocode in the first answer in this question
whats is the difference between train, validation and test set, in neural networks?

andrelucas
  • 678
  • 3
  • 10
  • Your answer was really useful. Now, if the full validation set is calculated at each epoch, lets say we have 70 samples to train, 20 for validation and 10 for testing. So an epoch would be the 70 training sets + the 20 validation sets, right? Now, in every epoch I'm going to have 70 different errors of training? and 20 of calidation? or do I also calc the mean for training error (mean of 70) and validation (mean of 20) and those would be my training error and validation error for one epoch? (I hope I am explaining ok my question) – gkapellmann Mar 09 '14 at 13:51
  • 3
    No. In one epoch, you have one training error, and one validation error. They will be different on each epoch. The training error is used to update the weights. The validation error is not. It can be used to exit training at a suitable point, usually where the two errors start to diverge. This is to avoid overfitting. Have a look again at the pseudocode in the link I gave. – andrelucas Mar 09 '14 at 16:43