Jaccard Distance calculation using pdist in scipy

Question

import scipy.spatial.distance as dist

Y=[[1,2,3],[2,3,4]]

Q=dist.pdist(Y,'jaccard')

print Q

The following snippet gives jaccard distance as 1 while it should be 0.5. On the other hand if Y=[[1,2,3],[4,2,3]] i.e if ordering is changed output is 0.33. But jaccard distance is independent of order of elements. Can you suggest how to resolve this issue here?

The [docs](http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.spatial.distance.pdist.html) aren't very clear, but they suggest that the ordering is important: they say that the Jaccard distance is "the proportion of those elements u[i] and v[i] that disagree", which I understand is for fixed i for both elements. That would agree with your results. Anyhow, did you check the implementation in their source code? — phfaist, Feb 20 '16 at 14:16
The docstring for the `jaccard` function (http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.jaccard.html) gives a better description. `jaccard` computes the Jaccard-Needham dissimilarity for *boolean* arrays. Its behavior for other array types is not defined, so you shouldn't be passing in arrays of arbitrary integers. — Warren Weckesser, Feb 20 '16 at 14:41

score 1 · Answer 1 · answered Feb 20 '16 at 23:21

The docstring for the jaccard function gives a better description of the calculation than the terse summary in the pdist docstring. jaccard computes the Jaccard-Needham dissimilarity for boolean arrays. Its behavior for other array types is not defined, so you shouldn't be passing in arrays of arbitrary integers.

score 0 · Answer 2 · edited Jan 02 '22 at 06:33

For anyone else with this issue, pdist appears to compare arrays by index rather than just what objects are present - so the scipy implementation is order dependent, but the input arrays are not treated as boolean arrays (in the sense that [1,2,3] and [4,5,6] are not both treated as [True True True], unlike the scipy jaccard function).

I had a similar issue and looked at it here:
Why are there discrepanices when generating a distance matrix with scipy pdist(metric = 'jaccard') vs scipy jaccard?

Jaccard Distance calculation using pdist in scipy

2 Answers2

Linked