Why do I get two different performances when creating Jaccard similarity matrix using two sparse matrices that seem to be the same

Question

I'm confounded by a strange performance issue when I try to create a Jaccard similarity matrix using sim2() from text2vec package. I have a sparse matrix [210,000 x 500] for which I'd like to obtain Jaccard similarity matrix as mentioned above. When I directly try to use the matrix in the sim2 function, it takes over 30 minutes and culminutes in error message

This is the R script I use:

library(text2vec)
JaccSim <- sim2(my_sparse_mx, method = "jaccard", norm = "none")  # doesn't work

This is the error message i get after half an hour of running the script:

Cholmod error 'problem too large' at file ../Core/cholmod_sparse.c, line 92.

However, when I subset another sparse matrix from the original matrix, using all the rows and run the script, it takes only 3 minutes and the Jaccard similarity matrix (which is a sparse matrix itself) is generated successfully.

spmx_1 <- Matrix(my_sparse_mx[1:210000], sparse = TRUE)
JaccSim <- sim2(spmx_1, method = "jaccard", norm = "none") #works!

This one runs successfully. What is going on here? all I'm doing is subsetting my sparse_matrix into another matrix (using all rows of the original matrix) and using the second sparse matrix.

To clarify, my_sparse_mx has 210,000 rows (i created it having that many rows using the following:

my_sparse_mx <-Matrix(0,nrow = 210000,ncol = 500,sparse = TRUE))

and then filled it up with 1's accordingly throughout some other process. Also, when I do nrows(my_sparse_mx) I still get 210,000.

I'd like to know why this is happening.

Answer 1

spmx_1 <- Matrix(my_sparse_mx[1:210000], sparse = TRUE)

says to take the first 210000 elements of my_sparse_matrix and turn it into another matrix. The result of this will have 210000 rows and 1 column.

You probably wanted

spmx_1 <- Matrix(my_sparse_mx[1:210000, ], sparse = TRUE)

with the comma.

Answer 2

sim2 function calculates pairwise jaccard similarity which means result matrix for your case will be 210000*210000. Sparsity of this resulting matrix depends on the data and for some cases won't be a problem. I guess for your case it is quite dense and can't be handled by underlying Matrix routines.

Your subsetting as mentioned above is not correct - you missed comma. So you subset just first 210000 elements.