Debugging Regression with TensorFlow Algorithm - Python

Question

Morning, I'm making some progress with TensorFlow, but it's still a struggle. I'm finding that a lot of the online examples don't run.

Anyway, I've tried to write some code to apply a multi-layer neural network to solve a regression problem. However, I only get zeros out. Is anyone able to help me find where I'm going wrong in my code and my understanding?

I'm running Tensorflow 0.12 with Python 3.5 in Windows 10

Many thanks

import tensorflow as tf
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Parameters
learning_rate = 0.001
training_epochs = 100
batch_size = 5000

# Network Parameters
n_hidden_1 = 256
n_hidden_2 = 10
n_input = 9
n_classes = 1
n_samples = dataVar.shape[0]

# TensorFlow Graph Input
x = tf.placeholder("float", [None, n_input])
if (n_classes > 1):
    y = tf.placeholder("float", [None, n_classes])
else:
    y = tf.placeholder("float", [None,])

# Create Multilayer Model
def multilayer_perceptron(x, weights, biases):
    '''
    x: Place holder for data input
    weights: Dictionary of weights
    biases: Dictionary of biases
    '''

    # First hidden layer with RELU activation
    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
    layer_1 = tf.nn.relu(layer_1)

    # Second hidden layer with RELU activation
    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
    layer_2 = tf.nn.relu(layer_2)

    # Last output layer with linear activation
    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']
    return out_layer

# weights and biases
weights = {
        'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),
        'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
        'out': tf.Variable(tf.random_normal([n_hidden_2, n_classes]))
}

biases = {
        'b1' : tf.Variable(tf.random_normal([n_hidden_1])),
        'b2': tf.Variable(tf.random_normal([n_hidden_2])),
        'out': tf.Variable(tf.random_normal([n_classes]))
}

# Construct Model
pred = multilayer_perceptron(x, weights, biases)


# Define loss and optimizer
cost = tf.reduce_mean(tf.square(pred - y))
#cost  = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, y))
optimizer = tf.train.GradientDescentOptimizer(learning_rate = learning_rate).minimize(cost)

# Initialize variables
init = tf.initialize_all_variables()

# RUNNING THE SESSION

# launch the session
sess = tf.InteractiveSession()


# Initialize all the variables
sess.run(init)

# Training Epochs
for epoch in range(training_epochs):
    # Start with cost = 0
    avg_cost = 0.0

    # Convert total number of batches to integer
    total_batch = int(n_samples/batch_size)

    # Loop over all batches
    for i in range(total_batch):
        # Grab the next batch of training data and labels
        ind = np.random.randint(0, high=dataVar_scaled.shape[0], size=(batch_size))
        batch_x = dataVar_scaled[ind,:]
        batch_y = depth[ind]

        # Feed dictionary for optimization and loss value
        _, c,p = sess.run([optimizer, cost,pred], feed_dict={x: batch_x, y: batch_y})

        # Compute average loss
        avg_cost += c/total_batch

    print("Epoch: {} cost = {:.4f}".format(epoch+1, avg_cost))

print("Model has completed {} Epochs of training".format(training_epochs))


prediction = tf.argmax(sess.run(pred, feed_dict={x:dataVar_scaled}),1).eval()
print(prediction)
plt.plot(prediction,depth,'b.')

score 0 · Accepted Answer · answered Feb 23 '17 at 17:50

0

I am not sure by regression, did you mean linear regression or logistic regression. But looking into your code, I believe you mean linear regression.

In case it is linear regression, your problem is in this line of code:

prediction = tf.argmax(sess.run(pred, feed_dict={x:dataVar_scaled}),1).eval()

You are taking the location of maximum (using tf.argmax()) value present in your code. Since n_classes = 1, there is only one class present in your output and it is always first which is why you are getting zero value. So change that line to

prediction = sess.run(pred, feed_dict={x:dataVar_scaled}).

Now you will get the value of the output and not the location of maximum value.

In case it is logistic regression, you are clearly having only one class. How can you do classification when you have outputs only as one class.

Hope this helps you.

answered Feb 23 '17 at 17:50

Prasad

5,946
3
30
36

Thanks for your quick response. Sorry, yes, I meant linear regression. I've changed that line of code, but I'm still getting every input equal to the same output. Is there something else I'm doing wrong? – jlt199 Feb 23 '17 at 17:57
I ran through your code with my change and my own input. It gave me a non-zero value. I didn't have your `dataVar` values. So I am sure, something is wrong in that variable. Ensure that they are not all zeros. Accept my answer if it has helped you. – Prasad Feb 23 '17 at 18:13
Thanks for your time. I don't think my input data is the problem, but I will double-check – jlt199 Feb 23 '17 at 18:22
Yeah, you need to check your input data. Within your present code, hard-code some value directly and then you will observe you will get non-zero value. – Prasad Feb 23 '17 at 18:25
I was getting a non-zero value, just the same value regardless of the input. I figured that I needed to take the transpose of `pred` then I get better results. See http://stackoverflow.com/questions/38399609/tensorflow-deep-neural-network-for-regression-always-predict-same-results-in-one – jlt199 Feb 24 '17 at 16:51
I am happy that you found out the problem. :) – Prasad Feb 24 '17 at 17:07

Debugging Regression with TensorFlow Algorithm - Python

1 Answers1