13

I just spent half an hour looking into a bug in statsmodels' SARIMAX functionality that I could finally trace back to the fact that numpy.int32 fails type checks for int.

>>> import numpy as np
>>> foo = np.int32(3)
>>> isinstance(foo, int)
False

Is there a way to circumvent this kind of issue without explicit type conversions? Should proper code even test for types and not check if a variable can safely be cast to a type?

Edit: My question is answered by an account of what technical limitations or design decisions are the cause of this behaviour and how to pythonically handle cases where both pure python's int and numpy int32 or int64 types might appear.

Neuneck
  • 294
  • 1
  • 3
  • 14
  • `` is not ``. So if you are working with `numpy` instead of checking for `isinstance(foo, int)`, check for `isinstance(foo, np.int32)`. – Ma0 Jan 26 '18 at 09:15
  • @Ev.Kounis What reason is there, not to inherit from pure python's int type, allowing `isinstance` to recognize the type? – Neuneck Jan 26 '18 at 09:28
  • I see more problems created than solved with that. This is all explained well in the proposed dupe. *My favorite answer is the second in vote-count*. I think we have to make an important distinction here between *integer numbers* as we, humans, know them and `integer types` as they are implemented in programming languages. You probably want to check for the 1st but actually check for the 2nd. – Ma0 Jan 26 '18 at 09:29
  • 2
    For reference: you can check against [numeric abstract base classes](https://docs.python.org/3/library/numbers.html#module-numbers). `isinstance(foo, numbers.Integral)` seems to work for numpy types too. – MB-F Jan 26 '18 at 10:37
  • `isinstance` works for classes in the `__mro__`. Could you check the class of the `item`? – hpaulj Jan 26 '18 at 12:23
  • @hpaulj Which `item` are you referring to? I'm afraid I can't follow your comment. – Neuneck Jan 26 '18 at 14:41
  • `np.int32.__mro__` lists the inheritance stack of that class. `isinstance` returns `True` for any class on that stack. `np.int32` and `int` only share `object`. `item` extracts a element from its `numpy` dtype, so `isinstance(np.int32(3).item(),int)` is True. – hpaulj Jan 26 '18 at 17:13

2 Answers2

10

__mro__ lists the inheritance stack of a class:

np.int32.__mro__
Out[30]: 
(numpy.int32,
 numpy.signedinteger,
 numpy.integer,
 numpy.number,
 numpy.generic,
 object)

int.__mro__
Out[31]: (int, object)

For a basic array:

x=np.array([1,2,3])    
x.dtype
Out[33]: dtype('int32')

isinstance of classes on this stack returns True:

isinstance(x[0], np.int32)
Out[37]: True    
isinstance(x[0], np.number)
Out[38]: True

int isn't on this stack:

isinstance(x[0], int)
Out[39]: False    
isinstance(x[0], object)
Out[40]: True

item extracts a value from its numpy wrapper:

isinstance(x[0].item(), int)
Out[41]: True

@kazemakase suggests using the numbers module:

isinstance(x[0], numbers.Integral)
Out[47]: True

edit

isinstance accepts a tuple of classes, so we can handle both the int and numpy cases with one test:

In [259]: isinstance(x[0], (int,np.integer))                                                           
Out[259]: True
In [260]: isinstance(x[0].item(), (int,np.integer))                                                    
Out[260]: True
hpaulj
  • 221,503
  • 14
  • 230
  • 353
9

Why should numpy.int32 descend from int? int is a specific class. It is one way of representing integers. That doesn't mean that every class that represents integers should descend from int. numpy.int32 has different semantics and different methods - for example, it has most of the functionality needed to operate like a 0-dimensional array - and inheriting from int isn't particularly useful for implementing numpy.int32.

On some builds of Python 2 (Windows only?), numpy.int32 actually will descend from int (which is also 32-bit on those builds), but I believe this design decision dates back to a time when int performed wraparound arithmetic like numpy.int32 instead of promoting to long on overflow, and when operator.index didn't exist. It was a more reasonable decision back then.

As for how to treat numpy.int32 like int, numbers.Integral does a sort of okay job, but the implementation relies on people explicitly register-ing their classes with numbers.Integral, and people often don't think to do that. NumPy didn't add the register calls until 2014, 6 years after numbers.Integral was introduced. Similar libraries like SymPy still don't have the calls.

I find operator.index to be a better check:

try:
    real_int = operator.index(some_intlike_thing)
except TypeError:
    # Not intlike.
    do_something_about_that()

operator.index is the hook an int-like class has to implement to make its instances usable as a sequence index. It's a stricter check than int(x), which would accept 3.5 and '3'. Since there's a concrete, easily noticeable impact if this hook is missing, it's more likely to be present than numbers.Integral support.

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • I had an environment with Linux + Python 2.7 (from source) + numpy 1.11.1 (from pip) where `numpy.int64` descended from `int`, but `numpy.int32` did not. – S. Kirby Apr 09 '20 at 08:05
  • **2014 was six years ago.** In 2020 and beyond, this answer is wrong to various degrees. [SymPy, for example, now registers its number types with the stdlib `numbers` API](https://github.com/sympy/sympy/blob/5d1b2c6f04f583ec0beeea4b8238d4151b497332/sympy/core/numbers.py#L3937) just like NumPy. This means that `isinstance(some_intlike_thing, numbers.Integral)` now suffices to test integer-ness for everything except probably broken third-party hack jobs that you shouldn't care about anyway... *because they're broken.* – Cecil Curry May 14 '20 at 05:12
  • 1
    @CecilCurry: Note the absence of anything like `numbers.Integral.register(Integer)` in that file. `sympy.Integer` *still* isn't registered with `numbers.Integral`, and `issubclass(sympy.Integer, numbers.Integral)` is *still* False. – user2357112 May 14 '20 at 05:18
  • This is most likely a bug. The `numbers.Rational.register(Integer)` line should probably be `numbers.Integral.register(Integer)`. Yet, the `numbers` module is older than Python 3.0, almost 12 years old by now, and adoption is still so terrible that this lack of registration passes essentially unnoticed. A **lot** more people care about indexing than the number of people that care about the `numbers` module. – user2357112 May 14 '20 at 05:30