1

I am trying to preform dimension reduction to 2 dimensions from a 64535 x 67 matrix to a 64535 x 2 matrix. I am not sure what the reasons are for why the fit transform calculation below is getting hung besides the fact this is a large matrix and a large reduction. Is mds not equipped to deal with such a large matrix/reduction? Is there a workaround to this?

temp = df.select_dtypes(include=[np.number])
norm = (temp - temp.mean())/temp.std()

mds = sklearn.manifold.MDS(n_components=2, eps=0)
data2d = mds.fit_transform(norm)
cap
  • 337
  • 3
  • 14

1 Answers1

2

MDS has O(N^3) complexity, it is probably not hanging but is still running. Check out section 3 of below paper. MDS will perform on large matrices just fine but it will take considerable time. I have no idea what you are trying to do with your dimensionality reduction, but if you are just looking for speed I would suggest some type of randomized projections (which still work well).

http://web.mit.edu/cocosci/Papers/nips02-localglobal-in-press.pdf

James Steele
  • 645
  • 1
  • 6
  • 22