Perceptron learns to reproduce just one pattern all the time

Question

This is rather a weird problem.

A have a code of back propagation which works perfectly, like this:

Now, when I do batch learning I get wrong results even if it concerns just a simple scalar function approximation.

After training the network produces almost the same output for all input patterns.

By this moment I've tried:

1. Introduced bias weights
2. Tried with and without updating of input weights
3. Shuffled the patterns in batch learning
4. Tried to update after each pattern and accumulating
5. Initialized weights in different possible ways
6. Double-checked the code 10 times
7. Normalized accumulated updates by the number of patterns
8. Tried different layer, neuron numbers
9. Tried different activation functions
10. Tried different learning rates
11. Tried different number of epochs from 50 to 10000
12. Tried to normalize the data

I noticed that after a bunch of back propagations for just one pattern, the network produces almost the same output for large variety of inputs.

When I try to approximate a function, I always get just line (almost a line). Like this:

Related question: Neural Network Always Produces Same/Similar Outputs for Any Input And the suggestion to add bias neurons didn't solve my problem.

I found a post like:

When ANNs have trouble learning they often just learn to output the
average output values, regardless of the inputs. I don't know if this 
is the case or why it would be happening with such a simple NN.

which describes my situation closely enough. But how to deal with it?

I am coming to a conclusion that the situation I encounter has the right to be. Really, for each net configuration, one may just "cut" all the connections up to the output layer. This is really possible, for example, by setting all hidden weights to near-zero or setting biases at some insane values in order to oversaturate the hidden layer and make the output independent from the input. After that, we are free to adjust the output layer so that it just reproduces the output as is independently from the input. In batch learning, what happens is that the gradients get averaged and the net reproduces just the mean of the targets. The inputs do not play ANY role.

Answer 1

My answer can not be fully precise because you have not posted the content of the functions perceptron(...) and backpropagation(...).

But from what I guess, you train your network many times on ONE data, then completely on ONE other in a loop for data in training_data, which leads that your network will only remember the last one. Instead, try training your network on every data once, then do that again many times (invert the order of your nested loops).

In other word, the for I = 1:number of patterns loop should be inside the backpropagation(...) function's loop, so this function should contain two loops.

---

EXAMPLE (in C#):

Here are some parts of a backpropagation function, I simplified it here. At each update of the weights and biases, the entire network is "propagated". The following code can be found at this URL: https://visualstudiomagazine.com/articles/2015/04/01/back-propagation-using-c.aspx

public double[] Train(double[][] trainData, int maxEpochs, double learnRate, double momentum)
{
    //...
    Shuffle(sequence); // visit each training data in random order
    for (int ii = 0; ii < trainData.Length; ++ii)
    {
        //...
        ComputeOutputs(xValues); // copy xValues in, compute outputs 
        //...
        // Find new weights and biases
        // Update weights and biases
        //...
    } // each training item
}

---

Maybe what is not working is just that you want to enclose everything after this comment (in Batch learn as an example) with a secondary for loop to do multiple epochs of learning:

%--------------------------------------------------------------------------
%% Get all updates