8

I running kmeans in matlab on a 400x1000 matrix and for some reason whenever I run the algorithm I get different results. Below is a code example:

[idx, ~, ~, ~] = kmeans(factor_matrix, 10, 'dist','sqeuclidean','replicates',20);

For some reason, each time I run this code I get different results? any ideas?

I am using it to identify multicollinearity issues.

Thanks for the help!

user1129988
  • 1,516
  • 4
  • 19
  • 32

3 Answers3

18

The k-means implementation in MATLAB has a randomized component: the selection of initial centers. This causes different outcomes. Practically however, MATLAB runs k-means a number of times and returns you the clustering with the lowest distortion. If you're seeing wildly different clusterings each time, it may mean that your data is not amenable to the kind of clusters (spherical) that k-means looks for, and is an indication toward trying other clustering algorithms (e.g. spectral ones).

You can get deterministic behavior by passing it an initial set of centers as one of the function arguments (the start parameter). This will give you the same output clustering each time. There are several heuristics to choose the initial set of centers (e.g. K-means++).

Ansari
  • 8,168
  • 2
  • 23
  • 34
  • Is there simple and effective deterministic way to initialize K-Means? – Royi May 20 '16 at 15:31
  • The most simple but naive solution would be to choose some centroids yourself. If you apply some heuristics to find "good" centroids to start with you have a deterministic k-mean. You might want to read this paper which uses *Nearest Neighbour Search* to build a deterministic k-mean algorithm: http://research.ijcaonline.org/volume63/number15/pxc3885541.pdf – Hans-Helge Aug 15 '17 at 08:26
  • But centroids are supposed to converge always to the same constellation. In theory, K-Means leads to the same set of centroids. What makes the final outcome random, in practice? – Leevo Nov 05 '20 at 09:32
6

As you can read on the wiki, k-means algorithms are generally heuristic and partially probabilistic, the one in Matlab being no exception.

This means that there is a certain random part to the algorithm (in Matlab's case, repeatedly using random starting points to find the global solution). This makes kmeans output clusters that are of good-quality-on-average. But: given the pseudo-random nature of the algorithm, you will get slightly different clusters each time -- this is normal behavior.

Rody Oldenhuis
  • 37,726
  • 7
  • 50
  • 96
  • Nitpicking: Note that in general, "heuristic" and "stochastic" are two different properties of an algorithm. A heuristic algorithm uses approximate (but, in general, deterministic) ratings for its decisions, while a stochastic algorithm uses (pseudo-)random numbers for decision making. Combinations are of course possible. – Florian Brucker Aug 29 '12 at 06:42
  • @FlorianBrucker True. And it's not nitpicking IMO :) – Rody Oldenhuis Aug 29 '12 at 06:44
  • Aren't centroids supposed to converge alwayt to the same constellation? In theory, K-Means leads to the same set of centroids. What makes the final outcome random, in practice? – Leevo Nov 05 '20 at 09:31
2

This is called initialization problem, as kmeans starts with random iniinital points to cluster your data. matlab selects k random points and calculates the distance of points in your data to these locations and finds new centroids to further minimize the distance. so you might get different results for centroid locations, but the answer is similar.

user14372
  • 1,409
  • 2
  • 11
  • 12