Mahout rowSimilarity

Question

I am trying to compute row similarity between wikipedia documents. I have the tf-idf vectors in format Key class: class org.apache.hadoop.io.Text Value Class: class org.apache.mahout.math.VectorWritable. I am following the quick tour of text analysis from here: https://cwiki.apache.org/confluence/display/MAHOUT/Quick+tour+of+text+analysis+using+the+Mahout+command+line

I created a mahout matrix as follows:

mahout rowid \
   -i wikipedia-vectors/tfidf-vectors/part-r-00000
   -o wikipedia-matrix

I got the the number generated rows and columns:

vectors.RowIdJob: Wrote out matrix with 4587604 rows and 14121544 columns to wikipedia-matrix/matrix

The matrix is of format Key class: class org.apache.hadoop.io.IntWritable Value Class: class org.apache.mahout.math.VectorWritable

I also have a docIndex document with the following format: Key class: class org.apache.hadoop.io.IntWritable Value Class: class org.apache.hadoop.io.Text

Then when I run the rowsimilarity job

mahout rowsimilarity 
   -i wikipedia-matrix/matrix 
   -o wikipedia-similarity 
   -r 4587604
   --similarityClassname SIMILARITY_COSINE
   -m 50
   -ess

I am getting the following error:

13/08/25 15:18:18 INFO mapred.JobClient: Task Id : attempt_201308161435_0364_m_000001_1, Status : FAILED
java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.mahout.math.VectorWritable
    at org.apache.mahout.math.hadoop.similarity.cooccurrence.RowSimilarityJob$VectorNormMapper.map(RowSimilarityJob.java:183)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:648)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
    at org.apache.hadoop.mapred.Child.main(Child.java:260)

Could someone please help me with the error? I am not sure from where is this org.apache.hadoop.io.Text when the input matrix is of format Key class: class org.apache.hadoop.io.IntWritable Value Class: class org.apache.mahout.math.VectorWritable

Thank you very much.

Best, Dragan

When I am running it on the local machine on a single node mode, I don't have this problem. Probably I should mention that the hadoop cluster runs on Cloudera 0.20.2-cdh3u6. The Mahout version on the cluster is mahout-0.5+9.11 (http://tinyurl.com/qhjl4xh). I tried to run the rowSimilarity against mahout-0.5 but it is not available in this version. It is only from 0.6. I tried with all versions starting from 0.6 to 0.9 but no success. The problem with the single node instance for me is that it is consuming a lot of disk space for the intermediate results (more than 100GB),which I don't have it — Dragan Milcevski, Aug 28 '13 at 19:33
Now I run it with the following command hadoop jar mahout-examples-0.9-SNAPSHOT.jar org.apache.mahout.math.hadoop.similarity.cooccurrence.RowSimilarityJob -i /user/dmilchev/wikipedia-matrix/matrix -o /user/dmilchev/wikipedia-similarity -r 4587604 --similarityClassname SIMILARITY_COSINE -m 50 -ess and I did not get any error. I guess my problem is solved. — Dragan Milcevski, Aug 29 '13 at 14:39

score 3 · Accepted Answer · answered Sep 03 '13 at 08:55

I solved it using the following command:

hadoop jar mahout-examples-0.9-SNAPSHOT.jar 
org.apache.mahout.math.hadoop.similarity.cooccurrence.RowSimilarityJob 
-i /user/dmilchev/wikipedia-matrix/matrix 
-o /user/dmilchev/wikipedia-similarity 
-r 4587604 --similarityClassname SIMILARITY_COSINE -m 50 -ess

and I did not get any error.

Mahout rowSimilarity

1 Answers1

Linked