How to get the Intersection and Union of two Series in Pandas with non-unique values?

Question

If I have 2 Series objects, like so: [0,0,1] [1,0,0] How would I get the intersection and union of the two? They only contain booleans which means they are non-unique values.

I have a large Boolean matrix. I've minhashed it and now I'm trying to find the false positives and negatives which I think means that I have to get the Jaccard similarity for each original pair.

For intersection, see [this answer](https://stackoverflow.com/a/18079695/2799941). — andrew_reece, Nov 18 '17 at 05:34
And if they are boolean then then Whats the use of intersection and union. Are you just trying to recreate `&` and `|`. Curious, why do you need intresection over booleans? — Bharath M Shetty, Nov 18 '17 at 05:38
Why? Whats the need for that? Can you specify that, so we can come with a much better answer for your actual problem. — Bharath M Shetty, Nov 18 '17 at 05:40
I have a large Boolean matrix. I've minhashed it and now I'm trying to find the false positives and negatives which I think means that I have to get the Jaccard similarity for each original pair. — user3927312, Nov 18 '17 at 05:43
Add the same in your question. Also add expected output for your current input. For jaccard similartiy take a look here http://scikit-learn.org/stable/modules/generated/sklearn.metrics.jaccard_similarity_score.html — Bharath M Shetty, Nov 18 '17 at 05:45

score 2 · Accepted Answer · answered Nov 18 '17 at 06:11

2

Since you say they are booleans use logical_and and logical_or of numpy or & and | on series i.e

y1 = pd.Series([1,0,1,0])
y2 = pd.Series([1,0,0,1])

# Numpy approach 
intersection = np.logical_and(y1.values, y2.values)
union = np.logical_or(y1.values, y2.values)
intersection.sum() / union.sum()
# 0.33333333333333331

# Pandas approach 
sum(y1 & y2) / sum(y1 | y2)
# 0.33333333333333331

answered Nov 18 '17 at 06:11

Bharath M Shetty

30,075
6
57
108

Thanks. Although this is the answer it's way too slow. – user3927312 Nov 18 '17 at 06:28
Whats the length of the arrays? – Bharath M Shetty Nov 18 '17 at 06:29
Each array is exactly 6000. And there are 1000 of them. – user3927312 Nov 18 '17 at 06:31
well numpy should take less time. It takes 16 micro secords for small array comparatively it should take less time. – Bharath M Shetty Nov 18 '17 at 06:32
I'm going to try and parallelize this. – user3927312 Nov 18 '17 at 08:48

How to get the Intersection and Union of two Series in Pandas with non-unique values?

1 Answers1