2

I have been trying to learn TensorFlow, I'm trying to modify a simple linear regression example and transform it into a polynomial regression by adding 2 more variables. It shouldn't be so hard but somehow when I execute the optimizer instead of diminishing the loss it makes it explode to infinity! I would appreciate any insight on what I'm doing wrong.

The loss grows instead of going down. 

import matplotlib.pyplot as plt
import numpy as np
import tensorflow as tf

num_puntos = 1000
conjunto_puntos = []
for i in xrange(num_puntos):
    x1= np.random.normal(0.0, 1.55)
    y1= np.sin(x1) * 4.1 + 0.3 + np.random.normal(0.0, 0.03)
    conjunto_puntos.append([x1, y1])
x_data = [v[0] for v in conjunto_puntos]  #x data
y_data = [v[1] for v in conjunto_puntos]  # target data (y data)
x_data2 = [v[0]**2 for v in conjunto_puntos] # x data squared
x_data3 = [v[0]**3 for v in conjunto_puntos] # x data to the power of 3

plt.plot(x_data, y_data, 'ro', label='Original data') #plot of the original data
plt.legend()
plt.show() 



W = tf.Variable(tf.random_uniform([1], -1.0, 1.0))  #variable 1
W1 = tf.Variable(tf.random_uniform([1], -1.0, 1.0)) # variable 2
W2 = tf.Variable(tf.random_uniform([1], -1.0, 1.0)) #variable 3
b = tf.Variable(tf.zeros([1])) # variable 4
y = W * x_data + b + W1 * x_data2 + W2 * x_data3 # my polinomial model
loss = tf.reduce_mean(tf.square(y - y_data)) # the loss function 
optimizer = tf.train.GradientDescentOptimizer(0.5) # the optimizer
train = optimizer.minimize(loss)  #the train function
init = tf.initialize_all_variables() #variable initialization
sess = tf.Session() #tf session
sess.run(init) #initialization of variables

for step in xrange(16): #I train for only 16 times
    sess.run(train) #execute the gradient decent optimizer once.
    plt.plot(x_data, y_data, 'ro') #plot the original data
    plt.plot(x_data, sess.run(W) * x_data + sess.run(b) + sess.run(W1) * x_data2 + sess.run(W2) * x_data3,'ro') #plot my model
    plt.xlabel('x')             
    plt.xlim(-8,6)
    plt.ylim(-8,6)
    plt.ylabel('y')
    plt.legend()
    plt.show()
    print(sess.run(loss))

sess.close()
Diego Orellana
  • 994
  • 1
  • 9
  • 20
  • 2
    Your code looks correct to me, so the exploding loss suggests that the learning rate you're using for the GradientDescentOptimizer might be too high. Have you tried a smaller learning rate (something like 0.005) or a smarter optimiser like Adam? – Avishkar Bhoopchand Sep 01 '17 at 16:58
  • Thank you Avishkar! why we have to do this? why it explodes with a big learning rate? – Diego Orellana Sep 01 '17 at 16:58
  • 2
    Having a learning rate that's too high can result in the optimiser "overshooting" at each step. This video by Andrew Ng explains it really well https://www.coursera.org/learn/machine-learning/lecture/3iawu/gradient-descent-in-practice-ii-learning-rate – Avishkar Bhoopchand Sep 01 '17 at 17:00
  • @AvishkarBhoopchand you saved me, sir. – Diego Orellana Sep 01 '17 at 17:02

0 Answers0