How to use iteration method with SGD in Encog?

Question

I am using stochastic gradient descent with Adam update to optimise a simple feed forward neural network in Encog (Java). Stochastic gradient descent class provides an iteration(count) method to perform a desired number of iterations. However, the documentation does not make clear whether an iteration stands for one epoch (i.e. a weight update after going through all the training data points) or just a weight update based on a single batch.

Moreover, by calling the method getIteration(), it turns out that iteration(count) is executing twice the number of desired iterations at each step of the while loop. The error is reproducible by running the following piece of code:

// Define network structure
BasicNetwork network = EncogUtility.
    simpleFeedForward(inputSize, numL1Neurons, numL2Neurons, outputSize, true);
network.reset();

// Stochastic Gradient Descent with Adam
final StochasticGradientDescent train = new StochasticGradientDescent(network, trainingSet);
System.out.println("Update rule: " + train.getUpdateRule().toString());
System.out.println("Batch size: " + train.getBatchSize());

// Count the number of iterations per epoch
int iterationsPerEpoch = (int) Math.ceil(trainingSet.size()/ (double) train.getBatchSize());
System.out.println("Iterations per epoch: " + iterationsPerEpoch);

// Training
int epoch = 0;
StringBuilder line = new StringBuilder();
while (!train.isTrainingDone()) {
   train.iteration(iterationsPerEpoch);
   epoch++;
   line.setLength(0);
   line.append("Epoch #");
   line.append(epoch);
   line.append(", Iteration #");
   line.append(train.getIteration());
   line.append(", Training Error: ");
   line.append(network.calculateError(trainingSet));
   line.append(", Validation Error: ");
   line.append(network.calculateError(validationSet));
   System.out.println(line.toString());
}
train.finishTraining();

Why does getIteration() return twice the number of iterations specified in iteration(count)? Which is the best way to run one epoch every iteration of the while loop?

How to use iteration method with SGD in Encog?

0 Answers0