MLlib collaborative filtering to generate Top N recommendations

Question

I was looking to find out a way to generate top n recommendations for all users using MLlib's ALS matrix factorization, but remained unsuccessful. Can anybody tell me does any such method exist?

Answer 1

here is my current approach which seems to be extremely slow:

Iterator<Rating> it = ratings.toLocalIterator();
        while (it.hasNext()) {
            int user = it.next().user();
            if (!userList.contains(user)) {
                Rating[] rat = model.recommendProducts(user, 10);
                for (Rating r : rat) {
                    list.add(user + "," + r.product() + "," + r.rating());
                }
                userList.add(user);
            }
        }

any efficient approach would be greatly appreciated.

Answer 2

Your structure above is the correct one for using Spark ALS. But you are asking a LOT of the cluster to recommend for all users - given if you have a smallish cluster to work on.

You should first determine: How long for ONE user? Then multiply that by # users - you will likely see you are asking too much.

There may may be cluster issues / suboptimal configuration settings and/or just in general an underpowered cluster versus your requirements.

Answer 3

You can use:

model.recommendProductsForUsers(int num)

This is available from spark 1.4 https://spark.apache.org/docs/latest/api/java/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.html#recommendProductsForUsers(int)