0

I'm a beginner and I'm trying to implement Backpropagation in C# for school purposes (so no tensorflow for now, we have to learn it manually). I have 64 nodes for input layer and 64 nodes for the output layer, somewhat an Autoencoder structure because we will be discussing MLP later on.

I'm calculating Delta Output as:

delta_out = (y_out) * (1 - y_out) * (desired - y_out)

I have tested my program to an XOR input/output scenario and it will correctly guess for this scenario but if I will put all the 64 nodes of input and output, then it will not give me the correct prediction (like 0% accuracy).

I'm also trying to total all of the delta_out abs(delta_out). For the XOR scenario, the absolute sum of delta_out is approaching zero as training progresses. But if I choose the 64 input and output test, then the absolute sum of all delta_out starts from a very small number and stays there.

For the XOR that's properly working (I have also tried OR and AND tests which just works fine), I used the following structure 2 nodes for input, 4 nodes for hidden, and 1 node for output.

For the 64 input and output, I have tested various numbers of nodes for the hidden layer, starting from 8 nodes to 128 nodes. If I use 64 or more nodes for the hidden layer, then the absolute sum of all the delta_out is near 0 even at the start and changes too slow.

I have also tested various learning rates (different learning rate for hidden and output layer). I tested from 0.1 to 0.75 but it seems it doesn't help for the 64 input/output that I'm supposed to accomplish. I have also changed the number of epochs from 100k to 500k but nothing seems to help.

Maybe I don't understand the Backpropagation concept well?

James Arnold
  • 698
  • 3
  • 9
  • 22
  • Question: nodes = neurons? Is that what you mean? And you are talking about a network with 64 inputs values and just one output layer with 64 neurons? Is that correct? What you are actually trying to make the network learn exactly? Why you need 64 output neuron if you are trying to simulate an XOR? Can you make your question a bit more precise? – Umberto Feb 01 '18 at 10:02
  • Hi Umberto... No, the XOR test is different from the 64 input/output I'm trying to achieve. The XOR (2 inputs and one output neuron) is fine when training. But the 64 input/output neuron I'm really trying to achieve doesn't learn properly i.e. the absolute sum of delta_out is a very small number and stays there as suppose to starting from a larger number then approaches to zero after training. – James Arnold Feb 02 '18 at 00:00
  • Just for me to understand... What is your delta_out? Is what you subtract (or add) to the weights during training (for example in gradient descent)? Is it possible for you to post a reduced version of your code so that I could have a look at it? Or maybe put it on github and post a link? – Umberto Feb 02 '18 at 11:32
  • Hi Umberto, the formula for delta_out is above. Delta out is used to adjust weights afterwards. – James Arnold Feb 03 '18 at 00:17

0 Answers0