3

I have a question about clustering using a distance matrix, but sparse.

Is there a sparse distance object format that does not expand the matrix and can work with the sparse representation?

Currently I'm doing the following

# read sparse matrix
sparse <- readMM('sparse-matrix')
distance <- as.dist(sparse)

sparse-matrix is already the correct distance matrix, which has NA's for entries that are not connected.

>sparse
[1,] . . .
[2,] 1 . .
[3,] 1 . .

> as.dist(sparse)
1 2
2 1  
3 1 0

But converting it with as.dist fails with

Error in asMethod(object) : negative length vectors are not allowed

Presumably, because it expands the matrix to a complete form. The matrix (NxN) size is N = 49281 This format is needed (dist object) by for example the hclust method

Similar Question without any answer on the R help list

Joey
  • 1,349
  • 14
  • 26

1 Answers1

-5

How would a distance matrix be sparse? There is a distance between each two objects, so it is actually a very dense matrix. However, a triangular matrix is sufficient to describe the mutual distances (as D = D'). This is actually the case for the objects produced by dist.

If the distance matrix is sparse because lots of objects are the same, then maybe you'd want to calculate the distance matrix only on unique objects.

cbeleites unhappy with SX
  • 13,717
  • 5
  • 45
  • 57
  • Okay, let's rephrase. The vectors that make the matrix are not very connected. So lot's of entries in the distance matrix should be NA. But as.dist turns it into a triangle matrix. The dist matrix is precomputed and I'm "just" trying to use that ... – Joey Apr 09 '13 at 20:06
  • @Joey: I still probably don't get it. If you calculate the distance of one object with some feature being `NA` to another object, is there any possibility that this distance is *not* `NA`? – cbeleites unhappy with SX Apr 09 '13 at 21:02
  • This applies, when you use for example the cosine distance and ignore values which are 1, eg orthogonal – Joey Apr 10 '13 at 10:08