5

I'm new to data science and am currently learning different techniques that I can do with Python. Currently, I'm trying it out with Spotify's API for my own playlists.

The goal is to find the most dissimilar features between two different playlist.

My question is what is the best way to identify the most dissimilar features between these two playlists?

I started off by getting all the tracks in each playlist and their respective features. I then computed the mean of each of the features.

Here is the DataFrame I ended up with. The data values are the means of all the tracks features to their respective playlist

                   playlist1  playlist2
                   --------------------
danceability      | 0.667509   0.592140
energy            | 0.598873   0.468020
acousticness      | 0.114511   0.398372
valence           | 0.376920   0.287250
instrumentalness  | 0.005238   0.227783
speechiness       | 0.243587   0.088612

I did some digging and found two common procedures:

1. Euclidean Distance

2. Cosine Similarity

I for some reason couldn't wrap my head around which one to use and proceeded to compute the absolute difference between each feature. Simple subtraction because that made sense to me intuitively. The feature with the greatest difference would be the 'most dissimilar'.

With this approach, I ended up using these results and concluded that energy and acousticness are the most dissimilar

                   playlist1    playlist2   absoluteDifference
                   ----------------------------------------------------
energy             |0.871310    0.468020    0.403290
acousticness       |0.041479    0.398372    0.356893
valence            |0.501890    0.287250    0.214640
instrumentalness   |0.049012    0.227783    0.178771
danceability       |0.531071    0.592140    0.061069
speechiness        |0.109587    0.088612    0.020975

Is my intuition correct/incorrect and when would we use the aforementioned techniques? Would any of those techniques be applicable in a situation such as this?

Eventually, I want to take the top two dissimilarities and make them my axis for KNN. My intuition is that I can identify the most dissimilar features of two playlists, I'll have a cleaner and more defined features of the playlist and can more accurately predict which song a playlist ought to belong to.

Mustafa
  • 337
  • 7
  • 14
  • 1
    Imo this is a question re method for calculating distance between two (or more?) points and as such is more suitable in the Data Science forum: https://datascience.stackexchange.com/. – user3471881 Nov 06 '18 at 15:34

1 Answers1

10

Let me start off with a few short remarks on Euclidean Distance and Cosine Similarity:

Euclidean Distance measures how far apart two points in a n-dimensional space are, i.e. it measures the length of a straight line from point A to point B

Cosine Similarity measures their similarity in orientation, i.e. the angle between two points A and B with vertex at zero

Let me add an image to underline my thoughts An illustration on different metrics The Euclidean distance between points A and B is depicted in red, the cosine similarity is depicted in green (and by that I don't literally mean the actual values of the measures, but rather what is relevant to their calculation)

Now let me talk about measures in general: any and all measures depict some kind of similarity. There is no such thing as a universal "best metric". The metric that suits your problem best is always determined by the problem.

I have added some extra points in the image to show that fact:

  • The points D and E have the same cosine similarity as A and B, but a vastly different Euclidean distance
  • On the contrary, the points A and F have a vastly different cosine similarity than A and B, but the same Euclidean distance

Now, let me make a remark to the appropriate choice of metric for your specific problem: you wish to evaluate how far apart features are. The bigger the difference, the farther apart the features. You don't care about angles between points at all. That is a clear point for Euclidean distance. You may not realize, but you actually used Euclidean distance in your example. Your features are one-dimensional, and in 1D, the Euclidean distance is equal to the absolute difference.

Lukas Thaler
  • 2,672
  • 5
  • 15
  • 31
  • Thank you for your post Lukas. It makes sense now. This brings up another question: Is Euclidean only used for 1D Array? Can we use Euclidean for multiple dimension? – Mustafa Nov 06 '18 at 22:01
  • 1
    Euclidean distance can be used for any number of dimensions, as it is the sum of point-wise squared differences (and a square root thrown on top of that). Let me elaborate (I'll use the 3-dimensional case, but it really works for any dimension): for two vectors (which represent points in higher dimensions) a = [1,1,3] and b = [5,4,3] the Euclidean distance is sqrt( (1-5)² + (1-4)² + (3-3)²) = sqrt(16 + 9) = 5. The higher dimensions, the longer the sum - but that's about all there is to Euclidean distance in higher dimensions – Lukas Thaler Nov 07 '18 at 12:25