I have programmed a simple NN library which creates a neural network with any size chosen and can train the network with a given activation function and its derivative. The networks do very well job with sigmoid as activation function but never converge with relu or leaky relu. I am using the problem of separating 2 groups of points on 2D grid as a standard problem to detect easily if the network is converging. This is the result of using sigmoid to solve the problem :: (x>0 and y>0) or (x<0 and y<0)
And this is the result of using Leaky Relu to solve the same problem:
I have implemented the predict and back propagate functions myself so i thought that there might be the problem but the fact that the network does well with sigmoid made me think that relu and leaky-relu are the problem. I have implemented them also myself but I can't see the problem in my implementation. Here is my Relu implemetation and its derivative:
## RelU
def Relu(num):
return np.where(num>0,num,0)
## Relu derivative
def Relu_deriv(num):
num=np.where(num>0,num,0)
num=np.where(num==0,num,1)
return num
Here is my Leaky relu implementation with its derivative:
## leaky RelU
def L_Relu(num):
return np.where(num>0,num,0.01*num)
## leaky RelU deriv
def L_Relu_D(num):
num=np.where(num<=0,num,1)
num=np.where(num>=0,num,0.01)
return num
I have also uploaded the code of the whole "library" to GitHub: Click to see the code in GitHub
You can see the code of the predict and back propagation functions in main.py. If you want to test the networks on the problem of separating two groups of points you can run the test functions in test_problem.py
I would really appreciate any advices about the reason behind the problem and how to solve it. Note: i though the problem might be dying relu but this should not happen with leaky relu.
