-2

I am training a neural network. For training I get 80% of my data and divide it to a number of mini-batches. I train on each mini batch, then update parameters, until all data is visited. I repeat the whole procedure for a number of epochs.

The question is about the remaining 10%+10% of data: how to change the validation set during this process? Should I use rotating mini batches for validation set as well?

OliverHennhoefer
  • 677
  • 2
  • 8
  • 21
user25004
  • 1,868
  • 1
  • 22
  • 47

1 Answers1

1

I think this question is more or less answered here: What is the meaning of batch_size for validation?

Since you don't train the model anymore - it does not affect the results. In other words, since you don't apply Mini-Batch Gradient Descent while validating your model with the validation set, it does not really matter. It may have an impact memory-wise though.

OliverHennhoefer
  • 677
  • 2
  • 8
  • 21
  • If while training, I want to print the validation error too to see the progress, should I compute on the complete validation set? – user25004 Aug 25 '22 at 18:43
  • To set hyperparameters, I should train on whole training set and validate on whole validation set> – user25004 Aug 25 '22 at 18:44
  • Yes you want the error of the complete validation set. To compute the error after each iteration/batch does not add any value in this case. – OliverHennhoefer Aug 25 '22 at 18:57
  • Can't we at least use it to decide about when to stop? – user25004 Aug 25 '22 at 19:10
  • 1
    Yes in case of hyperparameter tuning you may plot the training loss as well as the validation loss after each epoch to prevent overfitting for example ('learning curves'). So you train an epoch on the training data, calculate the training/validation loss with the current model state and repeat this procedure for n epochs. In my previous answer I solely focused on actual prediction via the model. – OliverHennhoefer Aug 25 '22 at 19:54
  • You could go through https://www.baeldung.com/cs/learning-curve-ml for more details – OliverHennhoefer Aug 25 '22 at 19:56