Spark ml cosine similarity: how to get 1 to n similarity score

Question

I read that I could use the columnSimilarities method that comes with RowMatrix to find the cosine similarity of various records (content-based). My data looks something like this:

genre,actor
horror,mohanlal shobhana pranav 
comedy,mammooty suraj dulquer
romance,fahad dileep manju
comedy,prithviraj

Now,I have created a spark-ml pipeline to calculate the tf-idf of the above text features (genre, actor) and uses the VectorAssembler in my pipeline to assemble both the features into a single column "features". After that, I convert my obtained DataFrame using this :

val vectorRdd = finalDF.map(row => row.getAs[Vector]("features"))

to convert it into an RDD[Vector]

Then, I obtain my RowMatrix by

val matrix = new RowMatrix(vectorRdd)

I am following this guide for a reference to cosine similarity and what I need is a method in spark-mllib to find the similarity between a particular record and all the others like this method in sklearn, as shown in the guide :

cosine_similarity(tfidf_matrix[0:1], tfidf_matrix)

But, I am unable to find how to do this. I don't understand what matrix.columnSimilarities() is comparing and returning. Can someone help me with what I am looking for?

Any help is appreciated! Thanks.

score 0 · Answer 1 · edited Sep 15 '17 at 05:20

I had calculated it myself with 2 small functions. Call cosineSimilarity on crossJoin of 2 dataframes.(seperate 1st line and others into 2)

def cosineSimilarity(vectorA: SparseVector, 
        vectorB:SparseVector,normASqrt:Double,normBSqrt:Double) :
    (Double,Double) = {
        var dotProduct = 0.0
        for (i <-  vectorA.indices){ 
            dotProduct += vectorA(i) * vectorB(i)
        }
        val div = (normASqrt * normBSqrt)
        if (div == 0 )
            (dotProduct,0)
        else
            (dotProduct,dotProduct / div)
    }

    val normSqrt : (org.apache.spark.ml.linalg.SparseVector => Double) = (vector: org.apache.spark.ml.linalg.SparseVector) => {
        var norm = 0.0
        for (i <- vector.indices ) {
            norm += Math.pow(vector(i), 2)
        }
        Math.sqrt(norm)
    }

Spark ml cosine similarity: how to get 1 to n similarity score

1 Answers1