17

There are two classes, let's call them X and O. A number of elements belonging to these classes are spread out in the xy-plane. Here is an example where the two classes are not linearly separable. It is not possible to draw a straight line that perfectly divides the Xs and the Os on each side of the line.

Members of two classes spread out on the xy-plane

How to determine, in general, whether the two classes are linearly separable?. I am interested in an algorithm where no assumptions are made regarding the number of elements or their distribution. An algorithm of the lowest computational complexity is of course preferred.

Raffael
  • 19,547
  • 15
  • 82
  • 160
Håvard Geithus
  • 5,544
  • 7
  • 36
  • 51

9 Answers9

17

If you found the convex hull for both the X points and the O points separately (i.e. you have two separate convex hulls at this stage) you would then just need to check whether any segments of the hulls intersected or whether either hull was enclosed by the other.

If the two hulls were found to be totally disjoint the two data-sets would be geometrically separable.

Since the hulls are convex by definition, any separator would be a straight line.

There are efficient algorithms that can be used both to find the convex hull (the qhull algorithm is based on an O(nlog(n)) quickhull approach I think), and to perform line-line intersection tests for a set of segments (sweepline at O(nlog(n))), so overall it seems that an efficient O(nlog(n)) algorithm should be possible.

This type of approach should also generalise to general k-way separation tests (where you have k groups of objects) by forming the convex hull and performing the intersection tests for each group.

It should also work in higher dimensions, although the intersection tests would start to become more challenging...

Hope this helps.

Darren Engwirda
  • 6,915
  • 4
  • 26
  • 42
  • 1
    This should do it. You can show that the optimal algorithm for taking convex hulls is O(n log n), since if it is not, then you can implement a sorting algorithm using the algorithm to compute convex hulls; and since you can not sort in faster than O(n log n), this implies you can not take convex hulls faster. – ldog Mar 20 '12 at 23:59
  • I should note, in any discussion of SVM's the intersection of convex hulls are very pertinent and usually occupy the first few paragraphs or chapters. – ldog Mar 21 '12 at 00:04
  • If one is implementing from scratch, it's easier to do O(n) point-in-polygon tests (O(log n)-time each) than line-line intersections: the hulls intersect if and only if they have vertices inside one another. This method is great for 2D but scales very poorly in d, the number of dimensions, because the convex hull can have size Omega(n^floor(d/2)). – rap music Mar 21 '12 at 13:00
  • 2
    I take back the great comment. This problem can be formulated as an LP with two variables and solved in time O(n) by the algorithm of Megiddo and Dyer. – rap music Mar 21 '12 at 13:08
  • @rapmusic: No. It's possible to have intersecting polygons where no vertices are enclosed. For instance, it's easy to draw two intersecting triangles where none of the vertices fall inside the other triangle. In general the intersection tests are necessary. – Darren Engwirda Mar 21 '12 at 20:43
  • @rapmusic: Could you provide a link to the LP method you've mentioned? – Darren Engwirda Mar 21 '12 at 20:50
6

Computationally the most effective way to decide whether two sets of points are linearly separable is by applying linear programming. GLTK is perfect for that purpose and pretty much every highlevel language offers an interface for it - R, Python, Octave, Julia, etc.

Let's say you have a set of points A and B:

enter image description here

Then you have to minimize the 0 for the following conditions:

(The A below is a matrix, not the set of points from above)

enter image description here

"Minimizing 0" effectively means that you don't need to actually optimize an objective function because this is not necessary to find out if the sets are linearly separable.

In the end (enter image description here) is defining the separating plane.

enter image description here

In case you are interested in a working example in R or the math details, then check this out.

Raffael
  • 19,547
  • 15
  • 82
  • 160
5

Here is a naïve algorithm that I'm quite sure will work (and, if so, shows that the problem is not NP-complete, as another post claims), but I wouldn't be surprised if it can be done more efficiently: If a separating line exists, it will be possible to move and rotate it until it hits two of the X'es or one X and one O. Therefore, we can simply look at all the possible lines that intersect two X'es or one X and one O, and see if any of them are dividing lines. So, for each of the O(n^2) pairs, iterate over all the n-2 other elements to see if all the X'es are on one side and all the O's on the other. Total time complexity: O(n^3).

Aasmund Eldhuset
  • 37,289
  • 4
  • 68
  • 81
  • Counterexample: 3 X in a small triangle, point down. 3 O widely spaced in a line below that. No line through any 2 X works. Now in that example if you instead search O it works, but I'm not sure if there's a further modification to invalidate that as well – Ben Jackson Mar 19 '12 at 23:35
  • @BenJackson: Whoops, forgot to include the modification I made to my own comment to your post... Fixed now. – Aasmund Eldhuset Mar 19 '12 at 23:47
4

Linear perceptron is guaranteed to find such separation if one exists.

See: http://en.wikipedia.org/wiki/Perceptron .

ElKamina
  • 7,747
  • 28
  • 43
  • I'm not familiar with this approach, but reading through the link you provided it suggests that the method would not converge for non-separable data sets. How can you deal with that case, and consequently, what's the overall complexity? – Darren Engwirda Mar 20 '12 at 07:19
  • @DarrenEngwirda You would use a Minimum Squared Error Gradient algorithm. `a_k+1 = a_k - p_k*Y'*(Y*a_k - b)` where Y is a matrix where the ith row is the ith sample point, p_k is p_1/k where p_1 is a positive constant, and b is an array of arbitrary scalars. – StockB Apr 23 '14 at 17:58
4

Computing a linear SVM then determining which side of the computed plane with optimal marginals each point lies on will tell you if the points are linearly separable.

This is overkill, but if you need a quick one off solution, there are many existing SVM libraries that will do this for you.

ldog
  • 11,707
  • 10
  • 54
  • 70
  • This is not really overkill, since this is equivalent to the linear programming technique that some other answers suggest. – Alexandre C. Sep 03 '15 at 07:27
4

You can probably apply linear programming to this problem. I'm not sure of its computational complexity in formal terms, but the technique is successfully applied to very large problems covering a wide range of domains.

High Performance Mark
  • 77,191
  • 7
  • 105
  • 161
  • 1
    In 2D, linear programming is linear-time by a result of Megiddo–Dyer. – rap music Mar 21 '12 at 13:05
  • And I would love to find out how ... http://math.stackexchange.com/questions/753976/how-to-check-linear-separability-of-points-using-linear-programming ... :) – Raffael Apr 15 '14 at 15:03
2

As mentioned by ElKamina, Linear Perceptron is guaranteed to find a solution if one exists. This approach is not efficient for large dimensions. Computationally the most effective way to decide whether two sets of points are linearly separable is by applying linear programming.

A code with an example to solve using Perceptron in Matlab is here

Rishi Dua
  • 2,296
  • 2
  • 24
  • 35
1

Well, both Perceptron and SVM (Support Vector Machines) can tell if two data sets are separable linearly, but SVM can find the Optimal Hiperplane of separability. Besides, it can work with n-dimensional vectors, not only points.

It is used in applications such as face recognition. I recomend to go deep into this topic.

Laurenz Albe
  • 209,280
  • 17
  • 206
  • 263
1

In general that problem is NP-hard but there are good approximate solutions like K-means clustering.

Ben Jackson
  • 90,079
  • 9
  • 98
  • 150
  • 1
    Maybe I'm missing something obvious, but can't this be solved in _O(n^3)_ by checking every pair of X'es and, for each pair, check if the line between them is a separating line? (Edit: Need to check every pair of one X and one O too.) – Aasmund Eldhuset Mar 19 '12 at 23:06
  • The Wikipedia entry links to references where it claims that even 2 clusters in Euclidean space is NP-hard. – Ben Jackson Mar 19 '12 at 23:17
  • 1
    @BenJackson: K-means clustering is a different problem, in which the requirement is that each element should belong to the cluster whose mean value the element is closest to, and where there is only one type of elements. The OP's problem is to partition a collection of two types of elements into two clusters that can be separated by a line. – Aasmund Eldhuset Mar 19 '12 at 23:25
  • 1
    @AasmundEldhuset: You are correct, I answered the question of classifying elements into two groups, while the question asks about pre-classified data. So my answer is wrong, but I'll leave it here since the discussion is informative. – Ben Jackson Mar 19 '12 at 23:30
  • @AasmundEldhuset: However I think your answer is also wrong, since I can construct cases where checking lines between points in X finds all lines that bisect O even though a line exists which does not. I haven't constructed a case where it fails if you check both X and O though. – Ben Jackson Mar 19 '12 at 23:32