How do I get backpropagation to work for a MLP? MATLAB

Question

I am trying to get an MLP to work. My goal is to get the net to predict output Yt when given Yt-1,Yt-2...,Yt-10. I've been using a generated dataset, which should be no trouble. My net will always output a straight line and will shift that line up and down. The line is always flat everywhere, it does not start to curve. My target is a portion of a sin(x) curve.

This is my code:

Dataset:

% Underlying function
syms f(a)
f(a)=sin(a/50);

% Data: lagged
x=[1:100+10]';
% add noise
y=eval(f(x))+0.01*randn(numel(x),1); 

% rescale dataset to [0,1]
y=(y-min(y))/(max(y)-min(y));

% Lagged predictor
X=[];
for L=1:10
    temp=lagmatrix(y,-L);
    X=horzcat(X,temp);
end

% train set size m
y_training=y(1:100);
X_training=X(1:100,:);

Training:

%% Network

% Initialize weights
W1=randn(10,10)/10; Wb1=randn(10,1)/10;
W2=randn(10,10)/10; Wb2=randn(10,1)/10;
W3=randn(1,10)/10;  Wb3=randn(1,1)/10;

% Activation function
syms f(z); 
f(z)=1/(1+exp(-z));
Df=diff(f);

% Net parameters
alpha=3;
lambda=0;

costs=[];
m=numel(y_training); % size of training set
no_iter=500; % number of iterations of grad descent 
p=1;


for j=1:no_iter

    %Initialize error
    deltaW1=zeros(size(W1));  deltaWb1=zeros(size(Wb1));
    deltaW2=zeros(size(W2));  deltaWb2=zeros(size(Wb2));
    deltaW3=zeros(size(W3));  deltaWb3=zeros(size(Wb3));

    JW=0;
    keep_output=[];

    for i=1:m

        % input layer
        a1=[X_training(i,:)']; y_true=y_training(i);

        % FP activations
        z2=[W1 Wb1]*[a1; 1]; a2=eval(f(z2));
        z3=[W2 Wb2]*[a2; 1]; a3=eval(f(z3));
        z4=[W3 Wb3]*[a3; 1]; a4=eval(f(z4)); 

        % BP individual errors 
        delta_a4= -(y_true-a4) .* eval(Df(z4));
        delta_a3= W3'*delta_a4 .* eval(Df(z3));
        delta_a2= W2'*delta_a3 .* eval(Df(z2));

        % DJDW for each parameter
        DJDW3=delta_a4*(a3)'; DJDWb3=delta_a4;
        DJDW2=delta_a3*(a2)'; DJDWb2=delta_a3; 
        DJDW1=delta_a2*(a1)'; DJDWb1=delta_a2;     

        % summing DJDW of each example
        deltaW1=deltaW1+DJDW1; deltaWb1=deltaWb1+DJDWb1;
        deltaW2=deltaW2+DJDW2; deltaWb2=deltaWb2+DJDWb2;
        deltaW3=deltaW3+DJDW3; deltaWb3=deltaWb3+DJDWb3;

        % Cost function
        JW_xy=(y_true-a4)^2; % single example
        JW=JW+JW_xy; % aggregate

        % store output and true values
        keep_output=[keep_output a4];

    end

    % update weights according to average cost of current iteration
    W1=W1-alpha*( (1/m)*deltaW1) ; Wb1=Wb1-alpha*( (1/m)*deltaWb1);
    W2=W2-alpha*( (1/m)*deltaW2) ; Wb2=Wb2-alpha*( (1/m)*deltaWb2);
    W3=W3-alpha*( (1/m)*deltaW3) ; Wb3=Wb3-alpha*( (1/m)*deltaWb3);   

    clf, plot(y_training), hold on
    plot(keep_output);
    drawnow

end

How do I get backprop to work? i.e. to train the net so that the output on the training dataset is close to the example given. — andrebeu, Nov 02 '15 at 22:01
That's not a proper question. Be more specific, tell us where you're stuck at, show us some right/wrong behavior, problems/solutions you encountered. — runDOSrun, Nov 04 '15 at 11:06
"My net will always output a straight line and will shift that line up and down. " But I think I solved it! — andrebeu, Nov 05 '15 at 14:33

score 0 · Answer 1 · answered Nov 05 '15 at 14:35

It only seemed that the backpropagation algorithm wasn't working. What I observed was a line shifting up and down with each iteration. The problem was I wasn't scaling the output.

Given that I scaled the input dataset, the output should also be scaled.

How do I get backpropagation to work for a MLP? MATLAB

1 Answers1