Non-vectorized Gradient Descent

Question

I have a bug in the following code, which is returning inf, inf for the Thetas.

def gradient_descent(x, y, t0, t1, alpha, num_iters):
for i in range(num_iters):
  t0_sum = 0
  t1_sum = 0
  for i in range(m_num): # I have a feeling that the following partial derivatives are wrong
    t0_sum += ((t1*x[i])+t0 - y[i])
    t1_sum += (((t1*x[i])+t0 - y[i])*(x[i]))
  t0 = t0 - ( alpha/m_num * (t0_sum) )
  t1 = t1 - ( alpha/m_num * (t1_sum) )
return t0, t1

Thanks

The variables x and y are lists of integers, if that makes any difference. — David Duffrin, Feb 24 '16 at 01:30

score 0 · Accepted Answer · answered Feb 24 '16 at 14:18

Your code is almost correct. The only thing: you used the same variable for both loops.

Just change i by j in the first loop and it will work.

For validation you can use the normal equation, which provides the best solution for the problem without using any loops.

Here is my code:

import numpy as np

def gradient_descent(x, y, t0, t1, alpha, num_iters):

    m_num = len(x);

    for j in range(num_iters):
        t0_sum = 0
        t1_sum = 0

        for i in range(m_num):
            t0_sum += ((t1*x[i])+t0 - y[i])
            t1_sum += (((t1*x[i])+t0 - y[i])*(x[i]))

        t0 = t0 - ( alpha/m_num * (t0_sum) )
        t1 = t1 - ( alpha/m_num * (t1_sum) )

    return t0, t1


def norm_equation(x, y):
    m = len(x);
    x = np.asarray([x]).transpose()
    y = np.asarray([y]).transpose()

    x = np.hstack((np.ones((m, 1)), x))

    t = np.dot(np.dot(np.linalg.pinv(np.dot(x.transpose(), x)), x.transpose()), y)
    return t


x = [6, 5, 8, 7, 5, 8, 7, 8, 6, 5, 5, 14]
y = [17, 9, 13, 11, 6, 11, 4, 12, 6, 3, 3, 15]

t0 = 0
t1 = 0
alpha = 0.008
num_iters = 10000

t0, t1 = gradient_descent(x, y, t0, t1, alpha, num_iters)
print("Gradient descent:")
print("t0 = " + str(t0) + "; t1 = " + str(t1))

print

t = norm_equation(x, y)
print("Normal equation")
print("t0 = " + str(t.item(0)) + "; t1 = " + str(t.item(1)))

Result:

Gradient descent:
t0 = 1.56634355366; t1 = 1.08575561307

Normal equation
t0 = 1.56666666667; t1 = 1.08571428571

Thank you so much! I can't believe it was such a simple mistake — David Duffrin, Feb 25 '16 at 00:06

Non-vectorized Gradient Descent

1 Answers1