I'm trying to do handwriting recognition with a single layer of perceptrons, using the MNIST database in Java. Each perceptron is assigned to recognise a digit (so 10 perceptrons). They are each trained randomly on each of 50,000 samples for a number of epochs. When testing, the perceptron with the highest rate is selected (ie the one that is most confident that it is correct).
Using this method I am consistently getting 93.5% accuracy. However, I think to improve I'll need to add hidden layers, and implement backpropogation. When using the Sigmoid (squashing) function to calculate on the forward pass of my single layer network, it works wonderfully. However when I change my backwards pass (learning) function to match back-propogation I get ~70% accuracy. Can someone check over my algorithms to make sure this is correct?
I got the algorithm from a few places, and I think it is the same. For example this one: http://www.webpages.ttu.edu/dleverin/neural_network/neural_networks.html
Note I am treating the last weight as a bias. The 'error' fed into the learn function is just the desired result minus the actual result. example is the same array as input - the greyscale of each pixel in the digit.
Forward pass:
public final double apply( double[] input ) {
double ret = 0;
for (int i=0; i<wei.length-1; i++){
ret = ret + input[i]*wei[i];
}
ret+=wei[wei.length-1];
//apply squashing function
ret = 1/(1+Math.exp(-ret));
return ret;
}
Learning Function (backwards Pass)
public final void learn( double error, double result, double[] example, double alpha ) {
for (int i=0; i<wei.length-1; i++){
//wei[i] = wei[i] + alpha*error*example[i]; //This was original learning function - it gives 93.5% accuracy
wei[i]=wei[i]+ alpha* error * example[i]*result*(1-result); //This line gives ~70% accuracy
}
//wei[wei.length-1] += alpha*error; //this line works for bias
wei[wei.length-1]=wei[wei.length-1]+ alpha* error * 1*result*(1-result); //again, this makes results inaccurate
}
