Mahout recommendation in implicit feedback situation

Question

I'm working with the Mahout framework in order to get recommendations in implicit feedback context using the well-known movielens dataset (ml-100k) that I have binarized considering 1 all the ratings equal to four or five, zero all the other. In this dataset there are five split, each of which divided in test set and training set as usually.

In the recommendation process I train the recommender using a simple GenericBooleanPrefUserBasedRecommender and the TanimotoCoefficientSimilarity as described in these lines of code:

    DataModel trainModel = new FileDataModel(new File(String.valueOf(Main.class.getResource("/binarized/u1.base").getFile())));
    DataModel testModel = new FileDataModel(new File(String.valueOf(Main.class.getResource("/binarized/u1.test").getFile())));
    UserSimilarity similarity = new TanimotoCoefficientSimilarity(trainModel);
    UserNeighborhood neighborhood = new NearestNUserNeighborhood(35, similarity, trainModel);

    GenericBooleanPrefUserBasedRecommender userBased = new GenericBooleanPrefUserBasedRecommender(trainModel, neighborhood, similarity);

    long firstUser = testModel.getUserIDs().nextLong(); // get the first user

    // try to recommender items for the first user
    for(LongPrimitiveIterator iterItem = testModel.getItemIDsFromUser(firstUser).iterator(); iterItem.hasNext(); ) {
        long currItem = iterItem.nextLong();
        // estimates preference for the current item for the first user
        System.out.println("Estimated preference for item " + currItem + " is " + userBased.estimatePreference(firstUser, currItem));

    }

When I execute this code, the result is a list of 0.0 or 1.0 which are not useful in the context of top-n recommendation in implicit feedback context. Simply because I have to obtain, for each item, an estimated rate which stays in the range [0, 1] in order to rank the list in decreasing order and construct the top-n recommendation appropriately.

So what's the problem with this code? Have I missed something or something was incorrect? Or maybe is the Mahout framework that doesn't provide a proper way of using binary feedback?

Thank you in advance,

Alessandro Suglia

Answer 1

If you want recommendations you are calling the wrong function. You have to call recommend

List<RecommendedItem> items = userBased.recommend(firstUser, 10);
for(RecommendedItem item : items) {
    System.out.println(item.getItemID()+" Estimated preference: "+item.getValue());
}

More information can be found at the javadocs: https://builds.apache.org/job/mahout-quality/javadoc/org/apache/mahout/cf/taste/recommender/Recommender.html

An extensive code example can be found here:

https://github.com/ManuelB/facebook-recommender-demo/blob/master/src/main/java/de/apaxo/bedcon/FacebookRecommender.java

Answer 2

If you are trying to evaluation the recommender offline and you are using the in-memory item or user based recommender then Mahout has an evaluation framework for this. It will split the data into training and test automatically and randomly. It trains on the training set and run an evaluation on the test set giving back several metrics.

Check out the "Evaluation" section at the bottom of the wiki page here: https://mahout.apache.org/users/recommender/userbased-5-minutes.html

Each run of this will yield slightly different results due to the random hold out set.

I would caution about doing this across different recommender algorithms since the test is only checking one against itself. To compare two algorithms or implementations is more complicated. Be sure to use exactly the same data, training and test splits, and even then the results are questionable until you do A/B user testing.

Update: offline you said you are using a particular evaluation system and can't use Mahout's--no matter. Here is how it's done:

You can remove some data from the dataset. So remove certain preferences. Then train and obtain recommendations for the user’s who had some data withheld. The test data has not been used to train and get recs so you then compare what users’ actually preferred to the prediction made by the recommender. If all of them match you have 100% precision. Note that you are comparing recommendations to actual but held-out preferences.

If you are using some special tools you may be doing this to compare algorithms, which is not an exact thing at all, no matter what the Netflix prize may have led us to believe. If you are using offline tests to tune a specific recommender you may have better luck with the results.

In one installation we had real data and split it into test and training by date. 90% of older data was used to train, the most recent 10% was used to test. This would mimic the way data comes in. We compared the recommendations from the training data against the actual preferences in the help-out data and used MAP@some-number-of-recs as the score. This allows you to measure ranking, where RMSE does not. The Map score led us to several useful conclusions about tuning that were data dependent.

http://en.wikipedia.org/wiki/Information_retrieval#Mean_average_precision