28

In this Coursera course for machine learning, it says gradient descent should converge.

I'm using Linear regression from scikit learn. It doesn't provide gradient descent info. I have seen many questions on StackOverflow to implement linear regression with gradient descent.

How do we use Linear regression from scikit-learn in real world? OR Why doesn't scikit-learn provide gradient descent info in linear regression output?

Mehdi Charife
  • 722
  • 1
  • 7
  • 22
Netro
  • 7,119
  • 6
  • 40
  • 58
  • 1
    One note is for LogisticRegression, it does provide an argument called `solver` where you can select which optimizer it will use. It will show debug information for the optimizer if you set `verbose=1`. http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression – David Maust Dec 26 '15 at 22:33

1 Answers1

48

Scikit learn provides you two approaches to linear regression:

  1. LinearRegression object uses Ordinary Least Squares solver from scipy, as LR is one of two classifiers which have closed form solution. Despite the ML course - you can actually learn this model by just inverting and multiplicating some matrices.

  2. SGDRegressor which is an implementation of stochastic gradient descent, very generic one where you can choose your penalty terms. To obtain linear regression you choose loss to be L2 and penalty also to none (linear regression) or L2 (Ridge regression)

There is no "typical gradient descent" because it is rarely used in practise. If you can decompose your loss function into additive terms, then stochastic approach is known to behave better (thus SGD) and if you can spare enough memory - OLS method is faster and easier (thus first solution).

R Sadeghi
  • 3
  • 2
lejlot
  • 64,777
  • 8
  • 131
  • 164