Getting incorrect values of theta while trying to implement stochastic gradient descent

Question

I am trying to implement Stochastic Gradient Descent algorithm for logistic regression. I have written a small train function whose job is to get the theta values / coefficients. But the values of theta come out to be incorrect and are same as the one initialised. I could not understand the reason for this. Is it not the correct way to implement stochastic gradient descent?

Here is the code I wrote for it:

train <- function(data, labels, alpha = 0.0009) {

  theta <- seq(from = 0, to = 1, length.out = nrow(data))

  label <- label[,shuffle]
  data <- data[,shuffle]

  for(i in seq(1:ncol(data))) {
    h = hypothesis(x, theta)
    theta <- theta - (alpha * ((h - y) * data[,i]))
  }
  return(theta)
}

Please note that, each column in the data frame is one input. There are 20K columns and 456 rows. So, 20K input values for training. The corresponding data frame named labels has the correct value for the input training data. So for example column 45 in data has its corresponding y value in column 45 of labels.

In the regression above, I am trying to train to predict between the label 1 and label 0. So labels is a data frame that comprises of 0 and 1.

De Novo · Accepted Answer · 2018-03-31T09:07:39.783

I can't debug this for you without a minimal, complete, and verifiable example, but I can offer you a tool to help you debug it:

add browser() in the body of your function like this:

train <- function(data, labels, alpha = 0.001) {
  browser()
  # ... the rest of your function

Call train with your data. This will open up a browser session. You can enter help (not the function, just help) to get the commands to navigate in the browser, but in general, use n and s to step through the statements (s will step into a nested function call, n will step over). If you do this in RStudio, you can keep an eye on your environment tab to see what the values for, e.g., theta are, and see a current traceback. You can also evaluate any R expression, e.g., tail(theta) in the executing environment. Q exits the browser.

I'd recommend exploring what hypothesis returns in particular (I'd be surprised if it's not almost always 1). But I think you have other issues causing the undesired behavior you described (the return value for theta isn't changing from its initial assignment).

EDIT:

Fix the typo: label should be labels each time.

Compare the sum of your return with the sum of theta as it is initialized, and you'll see that the return value is not the same as your initialized theta. Hope that helped!

Getting incorrect values of theta while trying to implement stochastic gradient descent

1 Answers1