0

The SVMlight site (in the FAQ) offers a script that computes the weight vector of a hyperlane. It says it "computes the weighted sum of the support vectors".

What does this mean? That is, what does the script do and what is the meaning and use of this weight vector?

Thanks in advance!

Cheshie
  • 2,777
  • 6
  • 32
  • 51
  • Could you point a more direct link to what are you referring to? I can't see the quoted text in the web site you've linked to. – BartoszKP Dec 29 '13 at 22:45
  • It's really weird, I can't (the Home page and FAQ have the same URL). If you CTRL+F "FAQ" on the home page - you'll find the link... and then CTRL+F the word "weight". Sorry about that... – Cheshie Dec 29 '13 at 22:47
  • Thanks, I've found it - but in the meantime lejlot has provided a good answer. – BartoszKP Dec 29 '13 at 22:48

2 Answers2

2

In linear case, the hyperplane can be always defined with d+1 numbers, where d is the dimension of the input space, while the number of actual support vectors may be much larger. By computing this hyperplane (lets call it w) you get more compact model, which can be then used to perform a classification:

cl(x) = sgn(w'x + b)

where w' is a transpositon of w

Things get much more tricky in the kernelized version, as w is in the form of the feature space projection, which may be unknown (or to expensive to compute) so one cannot get an equation of such an object (as it is no longer a hyperplane in the input space, but rather - a hyperplane in very rich feature space).

lejlot
  • 64,777
  • 8
  • 131
  • 164
  • Thanks @lejlot, but I didn't understand where the weight vector comes in... or do you mean that the script computes the values of 'w' from the support vectors (how?)? And does its values have any meaning? That is, if a certain feature received a zero, does that mean that it's unnecessary? – Cheshie Dec 29 '13 at 22:48
  • SVM solves the dual problem and look for alphas', and w=SUM_i y_i alpha_i sv_i, where sv_i is i'th support vector, y_i its class, and alpha_i its weight. this has the exact geometrical meaning, it is a normal to the separating hyperplane, and yes, if some feature has 0 weight it has no meaning. If it has a large value then it is a good predictor – lejlot Dec 29 '13 at 23:23
1

"Support vectors are the elements of the training set that would change the position of the dividing hyperplane if removed." The weights represent this hyperplane by providing the coordinates of a vector that is orthogonal to the hyperplane. "Computes the weighted sum of the support vectors" mathematically means sign(w'*x +b), when x is the support vectors and w' is the transpose of weight vectors, the value of w'x+b is 0 and it represents the decision boundary. When a new x reaches, the sign(w'x+b) will determine which class it belongs to.

For those x in the training sample that have the weight of 0, it means the sample does not contribute to the hyperplane, and including that x as a support vector will either increase the classification error, or decrease the margin.

Here is a reference tutorial with plenty of figures for more details.

lennon310
  • 12,503
  • 11
  • 43
  • 61