0

While trying to solve a problem, I came across the following limitation: numpy doesn't seem to have arbitrary integer precision!

Here is what I did:

import numpy as np

data = list(range(1, 21))

# ------- Regular Python --------
prod = 1
for el in data:
    prod*=el
print(prod)

# ------- NumPy --------
parr = np.prod(np.array(data, np.int64))
print(parr)

The results obtained are as follows:

2432902008176640000
2432902008176640000

So far, so good. Now, if we increase upper limit of the range from 21 to just 22, we can observe the limitation; the results will now be:

51090942171709440000
-4249290049419214848

My questions are

  1. why doesn't NumPy inherit the arbitrary integer precision from Python?
  2. why do we have this limitation? (or, am I making some mistake?)
  3. how to overcome it?

Update

The possible duplicate question suggests using a dtype=object, but, that takes away all numpy performance benefits for numeric data!

anurag
  • 1,715
  • 1
  • 8
  • 28
  • @mkrieger1 If I make a numpy array of object dtype, for integers, then it beats the purpose of using numpy (_as per the solution at the link you have suggested_) – anurag Feb 11 '21 at 21:12
  • Yes. But NumPy doesn't have an unlimited size integer type. If you want the NumPy performance you have to use the available types. – mkrieger1 Feb 11 '21 at 21:15
  • @mkrieger1 I figured as much, but, that is what confuses me - why can't numpy simply use the method Python handles it? I mean, I wish to understand the internal workings – anurag Feb 11 '21 at 21:16
  • 1
    AFAIK, numpy uses C backend for vectorization and acceleration. One thing about C acceleration is that one can perform arithmetic operations on larger memory block (128bits or so) using chip-specific instruction (e.g. SSE). That's where you need to know the specific memory layout, which pretty much dictates the `dtype` memory size. TLDR: it's would be difficult to write accelerated operators for arbitrary large integers – Quang Hoang Feb 11 '21 at 21:35
  • well, I guess the current solution is to use the `dtype=object` method. I guess we can close this question as a duplicate, then. Thanks and sorry – anurag Feb 11 '21 at 21:41
  • 2
    *Of course* `numpy` doesn't have arbitrary precision integers. The *whole point* of numpy is to use fixed-size arrays of primitive, numeric types *for their speed advantage* over Python objects. How did you expect `numpy` to have advantages over Python otherwise? You ask, "why can't numpy simply use the method Python handles it?" well, because *then it wouldn't be fast*, which is the whole point – juanpa.arrivillaga Feb 11 '21 at 23:31

0 Answers0