13

I'm new to SVMs in Matlab and need a little bit of help with it.

I want to train a support vector machine using the build in function fitcsvm of the Statistics Toolbox. Of course there are many parameter choices which control how the SVM will be trained.

The Matlab help is a litte bit wage about how the parameters archive a better training result. Especially the parameter 'Box Contraint' seems to have an important influence on the number of chosen support vectors and generalization quality.

The Help (http://de.mathworks.com/help/stats/fitcsvm.html#bt8v_z4-1) says

A parameter that controls the maximum penalty imposed on margin-violating observations, and aids in preventing overfitting (regularization).

If you increase the box constraint, then the SVM classifier assigns fewer support vectors. However, increasing the box constraint can lead to longer training times.

How exactly is this parameter used? Is it the same or something like the soft margin factor C in the Wikipedia reference? Or something completely different?

Thanks for your help.

Community
  • 1
  • 1
cahilx
  • 131
  • 1
  • 1
  • 3

1 Answers1

19

You were definitely on the right path. While description in the documentation of fitcsvm (as you posted in the question) is very short, you should have a look at the Understanding Support Vector Machines site in the MATLAB documentation.

In the non-separable case (often called Soft-Margin SVM), one allows misclassifications, at the cost of a penalty factor C. The mathematical formulation of the SVM then becomes:

SVM Minimization Problem

with the slack variables s_i which cause a penalty term which is weighted by C. Making C large increases the weight of misclassifications, which leads to a stricter separation. This factor C is called box constraint. The reason for this name is, that in the formulation of the dual optimization problem, the Langrange multipliers are bounded to be within the range [0,C]. C thus poses a box constraint on the Lagrange multipliers.

tl;dr your guess was right, it is the C in the soft margin SVM.

hbaderts
  • 14,136
  • 4
  • 41
  • 48
  • Thank you ... any hint on best value for box constraint? – user836026 Sep 02 '15 at 17:45
  • @user836026 It is not possible to have 1 best value for this. You will often end up tuning the box constraint to optimize the performance of your SVM. Some good details are given in the MATLAB article [Understanding SVMs](http://mathworks.com/help/stats/support-vector-machines-svm.html). – hbaderts Sep 14 '15 at 14:10
  • I tried different value for 'BoxConstraint' and I saw no changes in the result. – lenhhoxung May 07 '16 at 06:31
  • 1
    Nice description. A small error in the first formula: the constraint seems wrong, you want to increase the distance of your data x from the hyperplane w, so the inner product is not . Can you update the image you link to? – JStrahl Nov 09 '16 at 00:14
  • 1
    Oh, of course, you are right @JStrahl. I updated the equation accordingly. (Note: I created the equation using an [online LaTeX equation editor](http://www.codecogs.com/latex/eqneditor.php), so it is easy to re-create) – hbaderts Nov 09 '16 at 00:57
  • @hbaderts does it mean that we can set the box constraint parameter to determine the number of support vectors that are to be used in the formulation? I am getting confused between 'box constraint' and 'ShrinkagePeriod' parameters from the documentation. – roni Jun 21 '17 at 06:29
  • @roni a smaller box constraint allows for more points to be within the margin or even misclassified. This will most probably lead to *more* support vectors, i.e., yes you can control the number of support vectors with that. Though you can not specify e.g. "I want at most 1000 support vectors" with the box constraint. The `ShrinkagePeriod` parameter has nothing to do with the SVM itself, but only with the optimization: it gives you exactly the same number of support vectors, but may speed up convergence if you happen to have very few support vectors. – hbaderts Jun 21 '17 at 08:41
  • @hbaderts yes you are right ! From the documentation I found this to be written ... "Increasing BoxConstraint might decrease the number of support vectors, but also might increase training time...." But does this mean I should select a high value of box constraint as less number of points will then get misclassified i.e. improving the accuracy ? – roni Jun 21 '17 at 17:50