How To Do Linearly Separable Binary Classification?

Question

I want to solve following optimization problem -

Cost Function: 1/2 ||W||^2

Subject to : Y_i(w.X_i - b) >= 1

Where X is a 700x3 matrix, Y is a vector stores the label of classes for those instances (valued as 1/-1) and w.X_i is the dot product of w and X_i.

I am using CVX -

cvx_begin
    variable W(3);
    variable B;
    minimize (0.5*W'*W)
    subject to 
        Y'*(X*W - B) >= 1;
cvx_end

then, I am plotting, w1.x1 + w2.x2 - b which does not seem to be separating hyper-plane?

Whats wrong am I doing?

shouldn't a separating plane in this case have 3 variables only? — Palash Kumar, Feb 01 '15 at 13:55

score 0 · Answer 1 · answered Feb 01 '15 at 14:09

In short: when you are doing w1.x1 + w2.x2 - b you are trying to specify a hyperplane at a particular location, which is also the same as specifying a particular point on a vector. To do either in a 3D space you need to use all three dimensions, so: w1.x1 + w2.x2 +w3.x3 - b

In longer: When performing a linear classification such as this, the task can be viewed in two ways:

Finding a separating hyperplane such that all samples of one class are on one side, and all samples of the other class are on the other side.
Finding a projection of the multidimensional space which the samples are in, into a single dimensional line, such that there is a point on the line which clearly separates them.

These are identical tasks, since the single dimension in 2 is essentially how far each sample is from the separating hyperplane (and which side said sample is on). I find it helps to bear both of these viewpoints in mind, particularly since the separating hyperplane is the plane orthogonal to the single dimensional vector.

So, in the case you are dealing with, the weight vector w provided by the model is used to project the samples in matrix X onto a single dimensional line and the offset b indicates at which point along this vector the separating hyperplane occurs. By subtracting b from the projected values they are shifted such that this hyperplane is the one orthogonal to the line at point 0 which makes for simple thresholding.

How To Do Linearly Separable Binary Classification?

1 Answers1