Why's there a difference in prediction result between AdaBoost with n_estimators=1 that uses SVC as a base estimator, and just SVC

Question

I am currently using daily financial data to fit my SVM and AdaBoost. To check my result, I tried AdaBoost with n_estimators=1 so that it would return same result as I just run a single SVM.

from sklearn.ensemble import AdaBoostClassifier
from sklearn.svm import SVC


svm2 = SVC(C=box_const, kernel='rbf', degree=3, gamma='scale', coef0=0.0,
          shrinking=True, tol=0.001, cache_size=1000, class_weight='balanced',
          verbose=False, max_iter=-1, decision_function_shape='ovr', probability=True)
model2 = AdaBoostClassifier(base_estimator=svm2,
                           n_estimators=1,
                           algorithm='SAMME.R')


model2.fit(X_train, y_train)
svm2.fit(X_train, y_train)

However, on the contrary, I found that even though I set n_estimators=1, they produced different prediction result. Have I done something wrong? Or is there a specific reason to this result?

>>> model2.predict(X_test)
array([1., 1., 1., 1., 1.])
>>> model2.base_estimator
SVC(C=1, cache_size=1000, class_weight='balanced', probability=True)

>>> svm2.predict(X_test)
array([0., 1., 1., 0., 0.])
>>> svm2
SVC(C=1, cache_size=1000, class_weight='balanced', probability=True)

[Edit] I've found out that there is a significant difference regarding the ways I add sample_weight to the scikit learn's SVC.

When I define my model as such

svm2 = SVC(C=box_const, kernel='rbf', degree=3, gamma='scale', coef0=0.0,
          shrinking=True, tol=0.001, cache_size=1000, class_weight='balanced',
          verbose=False, max_iter=-1, decision_function_shape='ovr', probability=True)

These two yields the same prediction results

svm2.fit(X, y, sample_weight=[1] * len(X))
svm2.fit(X, y)

while

svm2.fit(X, y, sample_weight=[1 / len(X)] * len(X))

yields different results. I believe that since AdaBoost initializes the sample weights with 1 / len(X), this kind of problem occurs. Have I done something wrong in inserting sample weights to SVM?

score 1 · Accepted Answer · answered Mar 17 '21 at 13:54

This was answered over at stats.SE a few years ago:
Sample weights scaling in sklearn.svm.SVC
See also the documentation:

sample_weight : array-like of shape (n_samples,), default=None

Per-sample weights. Rescale C per sample. Higher weights force the classifier to put more emphasis on these points.

To summarize, sample weights are incorporated into the SVM algorithm in sklearn by modifying the regularization parameter C. You can compensate for AdaBoost by increasing the parameter C in your underlying SVC.

Thanks. This solution seems to solve the underlying problem I am having. — 배상일, Mar 17 '21 at 14:07

score 0 · Answer 2 · answered Mar 17 '21 at 07:59

0

You haven't done anything wrong. The classifier sets a new random state every time you run it. To fix that just set the random_state parameter to any value you like.

Eg:

AdaBoostClassifier(base_estimator=svm2,
                     n_estimators=1,
                     algorithm='SAMME.R',
                     random_state=21 
                   )

For more details check the official documentation

answered Mar 17 '21 at 07:59

Shaikh Abuzar

101
1
9

Hello. I've tried adding random_state, and the prediction values are still different. I am not using the probabilities of support vector machine, (I am using just the predicted class values), so all due respect, I believe adding random_state parameter is irrelevant to this matter. – 배상일 Mar 17 '21 at 08:16

Why's there a difference in prediction result between AdaBoost with n_estimators=1 that uses SVC as a base estimator, and just SVC

2 Answers2