Different versions of sklearn give quite different training results

Question

We upgraded our sklearn from the old 0.13-git to 0.14.1, and find the performance of our logistic regression classifier changed quite a bit. The two classifiers trained with the same data have different coefficients, and thus often give different classification results.

As an experiment I used 5 data points (high dimensional) to train the LR classifier, and the results are:

0.13-git:

clf.fit(data_test.data, y)
LogisticRegression(C=10, class_weight='auto', dual=False, fit_intercept=True,
intercept_scaling=1, penalty='l2', tol=0.0001)
np.sort(clf.coef_)
array([[-0.12442518, -0.11137502, -0.11137502, ..., 0.05428562,
0.07329358, 0.08178794]])

0.14.1:

clf1.fit(data_test.data, y)
LogisticRegression(C=10, class_weight='auto', dual=False, fit_intercept=True,
intercept_scaling=1, penalty='l2', random_state=None, tol=0.0001)
np.sort(clf1.coef_)
array([[-0.11702073, -0.10505662, -0.10505662, ..., 0.05630517,
0.07651478, 0.08534311]])

I would say the difference is quite big, in the range of 10^(-2). Obviously the data I used here is not ideal, because the dimensionality of features is much bigger than the number of entries. However, it is often the case in practice too. Does it have something to do with feature selection? How can I make the results the same as before? I understand the new results are not necessarily worse than before, but now the focus is to make them as consistent as possible. Thanks.

In your 0.13.1 you don't have `random_state = None`, is this a typo? — EdChum, Apr 18 '15 at 18:48
0.13 version does not have that parameter. In 0.14.1 I also tried to set random_state=0, and the result is the same as random_state=None. — ymeng, Apr 18 '15 at 18:51
It's worth checking the docs as there are sometimes bugfixes in different versions — EdChum, Apr 18 '15 at 18:52
I'm surprised this issue has not been widely discussed, since it is so obvious. The docs mentioned this: The underlying C implementation uses a random number generator to select features when fitting the model. It is thus not uncommon, to have slightly different results for the same input data. If that happens, try with a smaller tol parameter. I suppose it talks about inconsistency within the same version, but I do get the same coefficients if I use the same version to train. — ymeng, Apr 18 '15 at 19:01
That may explain why they introduced the `random_state` param in `0.14.0` as this caught me out when training RandomForest classifiers, hopefully one of the sklearn devs will confirm this is the case but I suspect this is what has happened — EdChum, Apr 18 '15 at 19:06
Some progress to report here: I found if I set class_weight=None, then the two versions have the same performance. It is a a problem only if class_weight='auto' — ymeng, Apr 20 '15 at 18:09
`'auto'` sounds dubious and may behave similar the `random_state` still good to know you've found something — EdChum, Apr 20 '15 at 18:11
Without having a fixed randomstate it can be quite challenging to track down what's going on. Do you get the same results in any of those two versions if you run it 2 times in a row? — , May 01 '15 at 06:32
Also, maybe you could try to set the random state "manually" at the beginning of your script, e.g.,`import numpy as np` `np.random.seed(0)` or so — , Jun 04 '15 at 04:15

score 2 · Accepted Answer · edited Jun 26 '21 at 12:29

From the release 0.13 changelog:

Fixed class_weight support in svm.LinearSVC and linear_model.LogisticRegression by Andreas Müller. The meaning of class_weight was reversed as erroneously higher weight meant less positives of a given class in earlier releases.

However, the update's description is for the version 0.13, not a higher version. You mention that you used the version 0.13-git, maybe you used a pre-release of the version 0.13 where the feature was not edited: this way, the update could make sense relatively to your problem.

By looking at your coefficients, they are lower in the new version, which makes a bit of sense with the update's description stating that weights were initially lowered.

You might want to change your new LogisticRegression(...)'s parameters and try to adjust things a bit.

Different versions of sklearn give quite different training results

1 Answers1