2

I have around 3000 datapoints in 100D that I project to 2D with t-SNE. Each datapoint belongs to one of three classes. However, when I run the script on two separate computers I keep getting inconsistent results. Some inconsistency is expected as I use a random seed, however one of the computers keeps getting better results (I use a macbook pro and a stationary machine on Ubuntu).

I use the t-SNE implementation from Scikit-learn. The script and data is identical, I've manually copied the folder to make sure. The relevant code snippet looks like this:

X_vectors, X_labels = self.load_data(spec_path, sound_path, subset)
tsne = TSNE(n_components=2, perplexity=25, random_state=None)
Y = tsne.fit_transform(X_vectors)
self.plot(X_labels, Y[:, 0], Y[:, 1], Y)

The first image is one sample generated from the macbook, I've ran it several times and it always generates a similar shape within the same x/y-range. The second is from Ubuntu and is clearly better, again I've ran it several times to make sure and it continues to generate better results, always in a higher x/y-range compared to the mac. Not sure what I'm not seeing here, it may very well be something obvious that I missed.

Result from the MacbookPro Result from Ubuntu

perfall
  • 118
  • 1
  • 9
  • 2
    Please use same integer value (not `None`) for random_state on both machines and then report the variations. – Vivek Kumar Oct 16 '17 at 11:02
  • It returns similar results as before. The reason I used None was to make sure that the fixed seed didn't have anything to do with it. – perfall Oct 16 '17 at 11:35

1 Answers1

5

TSNE is a heuristic. Like most heuristics, it might behave quite differently according to small changes. The core characteristic here is: only local-convergence is guaranteed! (not very robust). The latter is indicated (follows from basic optimization-theory) in the docs:

t-SNE has a cost function that is not convex, i.e. with different initializations we can get different results.

While you explained, that the non-seeding approach is not the culprit in your opinion (hard to measure! benchmarking is hard), you should check out your versions of sklearn, as the t-sne code is one of the more active parts of sklearn with many changes over time.

Each of these changes is likely to introduce observations like yours (when only trying one example; a bigger benchmark/testset should be a better approach when comparing t-sne implementations of course)

Remark: however one of the computers keeps getting better results: this is broad as there are at least two different interpretations:

  • rate the result visually / perceptually
  • look at kl_divergence_ achieved after optimization
sascha
  • 32,238
  • 6
  • 68
  • 110
  • 1
    A version update fixed it! The macbook had version 0.18.2 before, now both are 0.19 and they generate identical results given a fixed seed. Thank you.I agree that a "better result" is relative in this scenario, I was referring to non-overlapping clusters. – perfall Oct 16 '17 at 11:36
  • Thanks for the confirmation! – sascha Oct 16 '17 at 11:58