What is the correct approach to update weights vectors in neural network?

Question

I don't know how to update weights in Neural Network back-propagation. I train 18 tuples data. When should update the weight? After all tuples are calculated or one tuple is calculated? Please, help me. Thank in advance.

Answer 1

It depends. There are two approaches to learn weights in mathematical models (including neural networks) with differentiable [empirical] cost and internal functions. Both methods rely on computing gradients of the cost function with respect to the weights. The first method is known as full batch and the second mini-batch or stochastic gradient descent (SGD). However, for both methods one can take the derivative of the cost function single example (in your case tuple) with respect to a one weight only and update that weight, but this is not recommended as the weight update process can be done in parallel. So my suggestion is to use SGD and update the weights simultaneously.

Sources

• Stochastic Gradient Descent
• Batch Learning
• Learning Weights in Neural Networks