Numpy bug? (python3)

Question

import numpy as np
mainList = []
numpyArray0 = np.array([1,2,3])
numpyArray1 = np.array([4,5,6])
mainList.append(numpyArray0)
mainList.append(numpyArray1)

print("numpyArray0 in mainList:")
try:
  print(numpyArray0 in mainList)
except ValueError:
  print("ValueError")

print("numpyArray1 in mainList:")
try:
  print(numpyArray1 in mainList)
except ValueError:
  print("ValueError")

print("mainList in numpyArray0:")
try:
  print(mainList in numpyArray0)
except ValueError:
  print("ValueError")

print("mainList in numpyArray1:")
try:
  print(mainList in numpyArray1)
except ValueError:
  print("ValueError")

print(numpyArray1 in mainList)

So I have the above code basically it creates 2 numpy arrays inside a normal python list (mainList) and then it checks to see if those 2 arrays are inside the list. The code should output:

numpyArray0 in mainList:
True
numpyArray1 in mainList:
**True**
mainList in numpyArray0:
True
mainList in numpyArray1:
True
**True**

But instead of outputing the above it outputs the following:

numpyArray0 in mainList:
True
numpyArray1 in mainList:
ValueError
mainList in numpyArray0:
True
mainList in numpyArray1:
True
Traceback (most recent call last):
  File "/home/user/Documents/pythonCode/temp.py", line 31, in <module>
    print(numpyArray1 in mainList)
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Am I doing anything incorrectly? Note that I tried updating python, numpy, and my os (debian) before running the code.

Possible duplicate: https://stackoverflow.com/questions/5488307/numpy-array-in-python-list — DYZ, Aug 22 '18 at 22:33
I am confused about why the above code works with numpyArray0, but not with numpyArray1, but thank you for the link. — , Aug 22 '18 at 22:35
I understand; it is not really a dupe, rather a reference. I suggest you remove everything from your code from `print("numpyArray0:")` all the way down and keep `numpyArray0 in mainList` and `numpyArray1 in mainList` - to make the question clear and simple. — DYZ, Aug 22 '18 at 22:35
In general an `in` test is not reliable unless you are clearly comparing object ids (e.g. an object class without the compare method), or the compare method itself is reliable (for your purposes). — hpaulj, Aug 23 '18 at 03:42

score 3 · Answer 1 · answered Aug 22 '18 at 22:50

3

numpyArray0 in mainList calls list.__contains__. A list's __contains__ method calls PyObject_RichCompareBool for each element of the list to check if the elements are equal. As it happens, PyObject_RichCompareBool checks for identity equality first, and then does a full comparison.

numpyArray0 is mainList[0] returns True, so full comparison is never done. If full comparison was done, numpy would raise ValueError since a numpy array cannot be interepreted as a boolean.

numpyArray1 in mainList shows that as well (since identity comparison fails for numpyArray1 vs mainList[0].

answered Aug 22 '18 at 22:50

Alok Singhal

93,253
21
125
158

1

Why is `mainList in numpyArray0` then? And `mainList in numpyArray1`? – DYZ Aug 22 '18 at 22:53
As a further test, `numpyArray0[:] in mainList` raises the ValueError. The object id's no longer match, so it goes on to doing the element test. – hpaulj Aug 23 '18 at 03:38
1

@DYZ https://stackoverflow.com/questions/18320624/how-does-contains-work-for-ndarrays – Alok Singhal Aug 23 '18 at 04:05
Yeah, pretty surprising :). From my answer to that question: https://www.mail-archive.com/numpy-discussion@scipy.org/msg31578.html and https://github.com/numpy/numpy/issues/3016 are relevant. – Alok Singhal Aug 23 '18 at 04:07

score 1 · Accepted Answer · answered Aug 22 '18 at 22:51

1

Looks like it is a well-known feature related to the way the == operator is overloaded for Numpy arrays.

answered Aug 22 '18 at 22:51

DYZ

55,249
10
64
93

*This* is the right answer, *not* the accepted one. – MarianD Aug 23 '18 at 00:52
It's a combination of testing for id first, as described in @Alok's answer, followed by an array equality test - it's that equality test that produces the value error. – hpaulj Aug 23 '18 at 03:35

ShpielMeister · Answer 3 · 2018-08-22T23:08:26.270

0

here the gist:

>>> numpyArray1 in mainList
....    
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

this will work:

>>> any([list(numpyArray1) == list(litem)  for litem in mainList])

True

edited Aug 22 '18 at 23:08

answered Aug 22 '18 at 22:53

ShpielMeister

1,417
10
23

3

Your answer does not explain why `numpyArray0 in mainList` works. – DYZ Aug 22 '18 at 22:54
1

as @alok mentions it is essentially short circuited logic at work. since numpyArray0 is the first item in the mainList it is Ture because it is a single value. i.e. >>> numpyArray0 in [numpyArray0] True >>> numpyArray1 in [numpyArray1] True – ShpielMeister Aug 22 '18 at 23:03

score 0 · Answer 4 · answered Aug 23 '18 at 00:07

Compare the same operations with lists instead of numpy arrays:

In[171]: mainList = []
In[172]: list0 = [1,2,3]
In[173]: list1 = [4,5,6]
In[174]: mainList.append(list0)
In[175]: mainList.append(list1)
In[176]: list0 in mainList
Out[176]: True
In[177]: list1 in mainList
Out[177]: True

What am I trying to show with this?

Two things.

That the right answer of OP question gave DYZ, not Alok Singhal.
That the OP question is suspiciously similar to the reported bug. Compare with this part of reported issue in Python member operation does not work on lists because of overloaded ndarray equality #10400:

cache = []
cache.append(numpy.ndarray([1,2,3]))
cache.append(numpy.ndarray([4,5,6]))
numpy.ndarray([4,5,6]) in cache

"Bug" questions almost always get upvoted if they can be replicated. I imagine trawling bug reports on `github` and then making questions of them would get a new user some quick rep. But considering that even in that case, it still produces a useful question and answer I don't find it too disruptive. — Daniel F, Aug 23 '18 at 05:59

Numpy bug? (python3)

4 Answers4