I am trying to train a simple neural network with simulated annealing. I have programmed a neural network with an input layer of 784 input nodes (28 x 28 pixels: I am using the MNIST database to train), 1 hidden layer with 100 nodes and an output layer with 10 end nodes. I also programmed a simulated annealing algorithm that takes an input vector and minimizes a function to get the desired output vector.
Now my question is how to combine the two? I have read a couple papers but they don't specify exactly how this is done. I think the idea is as follows:
Initialize a vector of random weights (in my case the vector is of length 79,400; 78,400 weights for the input layer to the hidden layer, and 1,000 weights for the hidden layer to the output layer). Calculate the corresponding output, which will of course be incorrect, and the sum of squared errors. Then loop through the weight vector and adjust each weight slightly by adding or subtracting a small number. For each adjustment, calculate the sum of squared errors again and see which adjustment (adding or substracting) decreased this value. Repeating this process should in my opinion result in the weights that correspond to the desired output.
I would like to know if this approach is the way to go. If it is, this seems to be a very time consuming process and I was wondering whether there is a more efficient way to do this?
Thanks in advance!
