Back Propagation Neural Network Hidden Layer all output is 1

Question

everyone I have created a neural network with 1600 input, one hidden layer with different number of neurons nodes and 24 output neurons. My code shown that I can decrease the error each epoch, but the output of hidden layer always is 1. Due to this reason, the weight adjusted always produce same result for my testing data. I try different number of neuron nodes and learning rate in the ANN and also randomly initialize my initial weight. I use sigmoid function as my activate function since my output is either 1 or 0 in different output. May I know that what is the main reason that causes the output of hidden layer always is 1 and how should i solve it? My purpose for this neural network is to recognize 24 hand shape for alphabet, I try intensities data in my first phase of project. I have try 30 hidden neural nodes also 100 neural nodes even 1000 neural nodes but the output of hidden layer still is 1. Due to this reason, all of the outcome in testing data is always similar. I added the code for my network Thanks

g =  inline('logsig(x)');
[row, col] = size(input);
numofInputNeurons = col;

weight_input_hidden = rand(numofInputNeurons, numofFirstHiddenNeurons);
weight_hidden_output = rand(numofFirstHiddenNeurons, numofOutputNeurons);

epochs = 0;    
errorMatrix = [];

while(true)
    if(totalEpochs > 0 && epochs >= totalEpochs)
        break;
    end
    totalError = 0;
    epochs = epochs + 1;
    for i = 1:row
        targetRow = zeros(1, numofOutputNeurons);
        targetRow(1, target(i)) = 1;

        hidden_output = g(input(1, 1:end)*weight_input_hidden);
        final_output = g(hidden_output*weight_hidden_output);

        error = abs(targetRow - final_output);
        error = sum(error);
        totalError = totalError + error;

        if(error ~= 0)
             delta_final_output = learningRate * (targetRow - final_output) .* final_output .* (1 - final_output);
             delta_hidden_output = learningRate * (hidden_output) .* (1-hidden_output) .* (delta_final_output * weight_hidden_output');

            for m = 1:numofFirstHiddenNeurons
                for n = 1:numofOutputNeurons
                    current_changes = delta_final_output(1, n) * hidden_output(1, m);
                    weight_hidden_output(m, n) = weight_hidden_output(m, n) + current_changes; 
                end
            end

            for m = 1:numofInputNeurons
                for n = 1:numofFirstHiddenNeurons
                    current_changes = delta_hidden_output(1, n) * input(1, m);
                    weight_input_hidden(m, n) = weight_input_hidden(m, n) + current_changes;       
                end
            end
        end
    end

    totalError = totalError / (row);
    errorMatrix(end + 1) =  totalError;

    if(errorThreshold > 0 && totalEpochs == 0 && totalError < errorThreshold)
            break;
    end

 end

Answer 1

I see a few obvious errors that need fixing in your code:

1) You have no negative weights when initialising. This is likely to get the network stuck. The weight initialisation should be something like:

weight_input_hidden = 0.2 * rand(numofInputNeurons, numofFirstHiddenNeurons) - 0.1;

2) You have not implemented bias. That will severely limit the ability of the network to learn. You should go back to your notes and figure that out, it is usually implemented as an extra column of 1's inserted into input and activation vectors/matrix before determining the activations of each layer, and there should be a matching additional column of weights.

3) Your delta for output layer is wrong. This line

delta_final_output = learningRate * (targetRow - final_output) .* final_output .* (1 - final_output);

. . . is not the delta for the output layer activations. It has some extra unwanted factors.

The correct delta for logloss objective function and sigmoid activation in output layer would be:

delta_final_output = (final_output - targetRow);

There are other possibilities, depending on your objective function, which is not shown. You original code is close to correct for mean squared error, which would probably still work if you changed the sign and removed the factor of learningRate

4) Your delta for hidden layer is wrong. This line:

delta_hidden_output = learningRate * (hidden_output) .* (1-hidden_output) .* (delta_final_output * weight_hidden_output');

. . . is not the delta for the hidden layer activations. You have multiplied by the learningRate for some reason (combined with the other delta that means you have a factor of learningRate squared).

The correct delta would be:

delta_hidden_output = (hidden_output) .* (1-hidden_output) .* (delta_final_output * weight_hidden_output');

5) Your weight update step needs adjusting to match fixes to (3) and (4). These lines:

current_changes = delta_final_output(1, n) * hidden_output(1, m);

would need to be adjusted to get correct sign and learning rate multiplier

current_changes = -learningRate * delta_final_output(1, n) * hidden_output(1, m);

That's 5 bugs from looking through the code, I may have missed some. But I think that's more than enough for now.