0

I am trying to implement logistic regression using gradient descent to find the weights of a multivariate function given some data. So far I have come up with the following and the gradientDescent() function works using the meanSquareError() input function. 

import numpy as np

def logisticFunction(x, w):
    one = np.array([1])
    wTx = np.dot(np.transpose(x), w)
    exp = np.exp(wTx)
    add = np.add(one, exp)
    div = np.divide(one, add)
    return div

def logisticError(x, y, w):
    logistic = logisticFunction(x, w)
    sub = np.subtract(y, logistic)
    dot = np.dot(x, sub)
    return np.negative(dot)

def gradientDescent(x, y, foo, w0, step, eps_err):
    wPrev = w0
    error = foo(x, y, wPrev)
    wNext = np.subtract(wPrev, np.dot(step, error))
    while math.fabs(np.sum(np.subtract(wNext, wPrev))) >= eps_err:
        wPrev = wNext
        error = foo(x, y, wPrev)
        wNext = np.subtract(wPrev, np.dot(step, error))
    return wNext

def meanSquareError(x, y, w):
    Xw = np.dot(np.transpose(x), w)
    sub = np.subtract(y, Xw)
    dot = np.dot(x, sub)
    return np.multiply(np.array([-2]), dot)

x = np.array([[0.86,0.09,-0.85,0.87,-0.44,-0.43,-1.1,0.40,-0.96,0.17],
              [1] * 10])
#print np.transpose(x)
y = np.array([2.49,0.83,-0.25,3.10,0.87,0.02,-0.12,1.81,-0.83,0.43])
eps_err = np.array([0.01] * len(x))
#print logisticError(x, y, np.array([1,1]))
print gradientDescent(x,y,logisticError,np.array([1,1]),0.05,0.1)

When I use the logisticError() function, I receive an overflow error and this is because the logistic function does not seem to be converging with the gradient descent. I can seem to find any errors with normal online searches so any help is appreciated.

dinamix
  • 651
  • 8
  • 22
  • Yes, the wTx matrix keeps growing without apparent bound. What have you done to trace the logic and intermediate values in the other functions? You should have some root-cause attempts for this and show the traced results. See [here](https://ericlippert.com/2014/03/05/how-to-debug-small-programs/) for basics. – Prune Jan 24 '17 at 00:52
  • The sample I added results in overflow but when I use gradientDescent(x,y,meanSquareError,np.array([1,1]),0.05,0.1), this works with the correct result – dinamix Jan 24 '17 at 01:06

0 Answers0