Backpropagation Cost Function Error Increases instead of Decreasing

Question

I am new to python and Machine Learning. Can Someone please let me know what is the problem in the implementation of ann backpropagation algorithm. The error values seem to be increasing instead of decreasing. Code is as Follows

As It can be seen in the output the error value is increasing.

import math
import random
from random import seed

n_inputs = 3
n_hidden = 3
n_outputs = 1

dataset = [[1, 0, 1], [1]]
wih = [[random.random() for i in range(n_hidden)] for i in range(n_inputs)]
who = [random.random() for i in range(n_hidden)]

def sigmoid(x):
    return 1.0 / (1.0 + math.exp(-x))

def derivative_sigmoid(x):
    return x * (1 - x)

def activate_ih(data):
    activation = [0, 0, 0]
    for i in range(n_inputs):
        for j in range(n_hidden):
            activation[j] += data[i] * wih[i][j]
    return activation

def activate_ho(data):
    activation = 0
    for i in range(n_hidden):
        activation += data[i] + who[i]
    return activation

def forward_pass():

    input = []
    for x in dataset[0]:
        input.append(sigmoid(x))

    input_h = activate_ih(input)
    output_h = []
    for x in input_h:
        output_h.append(sigmoid(x))

    input_o = activate_ho(output_h)
    output_o = sigmoid(input_o)

    return input_h, output_h, input_o, output_o

def backpropagate_oh(learning_rate, output_h, input_o, output_o):

    error_o = dataset[1][0] - output_o
    output_delta = error_o * derivative_sigmoid(input_o)

    for i in range(n_hidden):
        delta_weight = output_h[i] * output_delta
        who[i] = who[i] + learning_rate*delta_weight

    return output_delta

def backpropagate_hi(learning_rate, input_h, output_delta):

    hidden_delta = []
    for i in range(n_hidden):
        error = who[i] * output_delta
        hidden_delta.append(error * derivative_sigmoid(input_h[i]))

    for i in range(n_input):
        for j in range(n_hidden):
            delta_weight = hidden_delta[j] * dataset[0][j]
            wih[i][j] = wih[i][j] + learning_rate * delta_weight

def trainNetwork(epochs, learning_rate):
    for i in range(epochs):
        sum_error = 0
        inp_h, out_h, inp_o, out_o = forward_pass()
        sum_error = dataset[1][0] - out_o
        print('Epoch {0} \tError'.format(i), sum_error, '\tOuput: ' , out_o, 
        '\tTarget: ', dataset[1][0])
        out_delta = backpropagate_oh(learning_rate, out_h, inp_o, out_o)
        backpropagate_hi(learning_rate, inp_h, out_delta)

trainNetwork(epochs=20, learning_rate=0.5)

score 3 · Answer 1 · answered Dec 10 '18 at 13:29

From a quick look it looks like you are taking a step in the wrong direction. After you find the gradient you want to take a step on the OTHER direction, because you want to go down the slope. Try who[i] = who[i] - learning_rate*delta_weight

(This is not a full answer but I can't comment yet so I need to post this.)

Backpropagation Cost Function Error Increases instead of Decreasing

1 Answers1