I am developing an algorithm based on gradient descent and I would like to add early stoping regularization. I have an objectif function,F, and I minimize it with respect to W. This is given in the code below:
Data : X_Train, Y_Train
t=1;
while (t < MaxIteration):
W = W - step * Grad (F,X,W).
loss(t) = computeLoss(X,Y,W);
end
Now I want to add early stoping regularization : this technique would consist in choosing the moment when it is necessary to stop during the optimization process (break the loop). How should I choose this moment? I have to test my model for each iteration on the validation data and create a history? What I'm trying to do is given below:
Data: X_Train, Y_Train, X_val, Y_val;
t=1;
maxIteration = 100;
models = array of size maxIteration
while (t < MaxIteration):
W = W - step * Grad(F,X,W).
loss(t) = computeLoss(X,Y,W);
models(t) = W;
t=t+1;
end
How do I choose the W model among all those I have stored?