23

Logistic regression class in sklearn comes with L1 and L2 regularization. How can I turn off regularization to get the "raw" logistic fit such as in glmfit in Matlab? I think I can set C = large number but I don't think it is wise.

see for more details the documentation http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression

Hanan Shteingart
  • 8,480
  • 10
  • 53
  • 66
  • 1
    Yes, you can use `l2` penalty and set the `C` parameter large. How beneficial is it not to penalize? If you do this with a completely separable dataset, then the weights will diverge. – eickenberg Aug 22 '14 at 10:01
  • 1
    its beneficial if you want the maximum likelihood solution without regularization, for example if you have a lot of data and you don't worry about over fitting. – Hanan Shteingart Aug 24 '14 at 11:32
  • 1
    Is there a well-tested Python package that does regular logistic regression? I feel like this is what @HananShteingart is looking for. – Dylan Daniels May 19 '16 at 18:18
  • 2
    I believe the statsmodels API does logistic regression without regularization - see https://stats.stackexchange.com/questions/203740/logistic-regression-scikit-learn-vs-statsmodels – elz Aug 18 '17 at 18:52

3 Answers3

9

Yes, choose as large a number as possible. In regularization, the cost function includes a regularization expression, and keep in mind that the C parameter in sklearn regularization is the inverse of the regularization strength.

C in this case is 1/lambda, subject to the condition that C > 0.

Therefore, when C approaches infinity, then lambda approaches 0. When this happens, then the cost function becomes your standard error function, since the regularization expression becomes, for all intents and purposes, 0.

Update: In sklearn versions 0.21 and higher, you can disable regularization by passing in penalty='none'. Check out the documentation here.

Yu Chen
  • 6,540
  • 6
  • 51
  • 86
  • How large does C need to be if I want to use vanilla logistic regression? – haneulkim May 09 '21 at 02:59
  • @haneulkim in older versions of sklearn, you could not blanket disable regularization, so we'd set C to some large parameter like `1e9`. But I believe since version 0.21, you can pass in `penalty='none'`. – Yu Chen Aug 08 '21 at 02:09
6

Go ahead and set C as large as you please. Also, make sure to use l2 since l1 with that implementation can be painfully slow.

Phillip Chilton Adkins
  • 101
  • 1
  • 5
  • 1
    I have tried C=1e42, why does `l1` and `l2` still give different coefficients for each feature? I have about 100 features, though their coefficients from `l1` and `l2` are highly linearly correlated, but still far from being the same. I thought without regulariztion, the two should return exactly the same coefficients. Do you have any idea what causes the difference, please? – zyxue Nov 25 '16 at 16:58
-2

I got the same question and tried out the answer in addition to the other answers:

If set C to a large value does not work for you, also set penalty='l1'.

Code Learner
  • 191
  • 11