Pearson vs Euclidean vs Manhattan Results

Question

Using Python 3.6. I am not getting logical results when using Manhattan distance for similarity measurement. Even comparing to the results from Pearson and Euclidean correlation, the units for Euclidean and Manhattan looks off?

I am working on a crude recommendation model that involves recommending similar items by measuring similarity between user rating for preferred item X and other user ratings for the same item, and recommending the items of the other users with whom a strong match is found with the user who raised the request

The results I got are

Pearson: 
 [('Men in Black II', 0.12754201365635218), ('Fried Green Tomatoes', 0.11361596992427059), ('Miami Vice', 0.11068770878125743), ('The Dark', 0.11035867466994702), ('Comanche Station', 0.10994620915146613), ('Terminator 3: Rise of the Machines', 0.10802689932238932), ('Stand by Me', 0.10797224471029637), ('Dancer in the Dark', 0.10241410378191894), ('Los Olvidados', 0.10044018848844877), ('A Shot in the Dark', 0.10036315249837004)]

Euclidean: 
 [('...And the Pursuit of Happiness', 1.0), ('12 Angry Men', 1.0), ('4 Little Girls', 1.0), ('4교시 추리영역', 1.0), ('8MM', 1.0), ('A Band Called Death', 1.0), ('A Blank on the Map', 1.0), ('A Dandy in Aspic', 1.0), ('A Date with Judy', 1.0), ('A Zona', 1.0)]

Manhattan: 
 [('...And the Pursuit of Happiness', 1.0), ('12 Angry Men', 1.0), ('4 Little Girls', 1.0), ('4교시 추리영역', 1.0), ('8MM', 1.0), ('A Band Called Death', 1.0), ('A Blank on the Map', 1.0), ('A Dandy in Aspic', 1.0), ('A Date with Judy', 1.0), ('A Zona', 1.0)]

Cosine: 
 [('...And the Pursuit of Happiness', 1.0), ('4 Little Girls', 1.0), ('4교시 추리영역', 1.0), ('8MM', 1.0), ('A Band Called Death', 1.0), ('A Blank on the Map', 1.0), ('A Dandy in Aspic', 1.0), ('A Date with Judy', 1.0), ('A Zona', 1.0), ('A.I. Artificial Intelligence', 1.0)]

Answer 1

I cannot tell you why you get strange results without seeing your code, however, I can give you some explanation of the difference between Pearson, Euclidean and Manhattan similarities between two vectors.

1. Pearson: this can be thought of as the cosine between the two vectors, and is therefore scale invariant. Thus, if two vectors are the same, but scaled diffently, this will be 1. With movie recommendations I assume this means that if I rated movie 1: 2/5, movie 2: 1/5 and movie 3: 2/5 and you rated the same movies 4/5, 2/5 and 4/5 respectively, then we will have the same movies recommended to us.

2. Euclid: This is the normal way to measure distance between vectors. Note that large differences are exaggerated and small differences are ignored (small numbers squared becomes tiny numbers, large numbers squared becomes huge numbers). Thus if two vectors almost agree everywhere, they will be regarded as very similar. Additionally scale matters, and the example above would give a relatively large dissimilarity.

3. Manhattan: This is similar to Euclidean in the way that scale matters, but differs in that it will not ignore small differences. If two vectors almost agree everywhere, the Manhattan distance will be large. Additionally, large differences in a single index will not have as large an impact on final similarities as with the Euclidean distance.

I assume that it is the fact that small dissimilarities add up to become a large dissimilarity in Manhattan but not Pearson and Euclidean that is the source of your confusion.

Ok, so upon looking at your code some more, I see that you use 1/(1+euclidean_distance) for Euclidean similarity, but manhattan_distance for Manhattan similarity. Try this instead

def Manhattan(x, y):
    return 1/(1+np.sum(np.abs(x-y)))

Ps. Sorry for any typos, I'm on my phone. Hopefully everything is still understandable.

Pps. note that you can write np.linalg.norm(x-y) for Euclidean distance between x and y and np.linalg.norm(x-y, 1) for Manhattan distance between x and y (instead of dealing with sqrt(sum((x-y)**2)) and np.sum(np.abs(x-y)).