12

On my Anaconda Python distribution, copying a Numpy array that is exactly 16 GB or larger (regardless of dtype) sets all elements of the copy to 0:

>>> np.arange(2 ** 31 - 1).copy()  # works fine
array([         0,          1,          2, ..., 2147483644, 2147483645,
       2147483646])
>>> np.arange(2 ** 31).copy()  # wait, what?!
array([0, 0, 0, ..., 0, 0, 0])
>>> np.arange(2 ** 32 - 1, dtype=np.float32).copy()
array([  0.00000000e+00,   1.00000000e+00,   2.00000000e+00, ...,
         4.29496730e+09,   4.29496730e+09,   4.29496730e+09], dtype=float32)
>>> np.arange(2 ** 32, dtype=np.float32).copy()
array([ 0.,  0.,  0., ...,  0.,  0.,  0.], dtype=float32)

Here is np.__config__.show() for this distribution:

blas_opt_info:
    library_dirs = ['/users/username/.anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/users/username/.anaconda3/include']
    libraries = ['mkl_rt', 'pthread']
lapack_opt_info:
    library_dirs = ['/users/username/.anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/users/username/.anaconda3/include']
    libraries = ['mkl_rt', 'pthread']
mkl_info:
    library_dirs = ['/users/username/.anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/users/username/.anaconda3/include']
    libraries = ['mkl_rt', 'pthread']
openblas_lapack_info:
  NOT AVAILABLE
lapack_mkl_info:
    library_dirs = ['/users/username/.anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/users/username/.anaconda3/include']
    libraries = ['mkl_rt', 'pthread']
blas_mkl_info:
    library_dirs = ['/users/username/.anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/users/username/.anaconda3/include']
    libraries = ['mkl_rt', 'pthread']

For comparison, here is np.__config__.show() for my system Python distribution, which does not have this problem:

blas_opt_info:
    define_macros = [('HAVE_CBLAS', None)]
    libraries = ['openblas', 'openblas']
    language = c
    library_dirs = ['/usr/local/lib']
openblas_lapack_info:
    define_macros = [('HAVE_CBLAS', None)]
    libraries = ['openblas', 'openblas']
    language = c
    library_dirs = ['/usr/local/lib']
openblas_info:
    define_macros = [('HAVE_CBLAS', None)]
    libraries = ['openblas', 'openblas']
    language = c
    library_dirs = ['/usr/local/lib']
lapack_opt_info:
    define_macros = [('HAVE_CBLAS', None)]
    libraries = ['openblas', 'openblas']
    language = c
    library_dirs = ['/usr/local/lib']
blas_mkl_info:
  NOT AVAILABLE

I'm wondering if the MKL acceleration is the problem. I've reproduced the bug on both Python 2 and 3.

1''
  • 26,823
  • 32
  • 143
  • 200
  • First question: Why? :-) ... Could you provide some more details: What is the `dtype` of your arrays before/after the copy? Does `np.arange(2 ** 31)` work correctly? What about `np.arange(1, 2**31)` and `np.arange(1, 2**31).copy()`? What is the dtype of `np.intp` on your computer? – MSeifert Feb 05 '17 at 07:30
  • @MSeifert The dtype is `float64` for the first two examples and `float32` for the second two. `np.arange(2 ** 31)` and `np.arange(1, 2**31)` both work correctly - it seems to be an issue with the copying. `np.arange(1, 2**31).copy()` also works, probably because it's under 16 GB. `np.intp` is `int64` (I'm running 64-bit Python and OS). – 1'' Feb 05 '17 at 07:34
  • 1
    Zero outputs sound like something that developers fall back to when things go wrong. It can also be interpreted null in C. I believe you should file a bug report. – Patrick the Cat Feb 05 '17 at 07:50
  • The MKL shouldn't have anything to do with it. That is only used for the linear algebra routines, not `ndarray.copy()`. What are the versions of the numpy that works and the numpy that doesn't? I vaguely recall a bug like this coming up some time ago, but then it was fixed. – Robert Kern Feb 05 '17 at 07:59
  • @RobertKern: They're both 1.11.2 - should I file a bug report or was it fixed more recently than this? – 1'' Feb 05 '17 at 08:07
  • If you can, try building numpy from source (at least the latest release 1.12.0) and check that. If you see the error there, report it to numpy. Otherwise, report the bug to Continuum. The system's version of numpy may have had patches applied. You can try to check your system's source package of numpy and see what patches they applied. – Robert Kern Feb 05 '17 at 08:16
  • @RobertKern: I installed the same 1.11.2 version inside the Anaconda distribution with `pip install --upgrade --no-deps numpy==1.11.2` and it doesn't have the bug. `np.__config__.show()` is the same as for the system Python. The latest 1.12.0 version doesn't have the bug either when installed this way. – 1'' Feb 05 '17 at 08:33

1 Answers1

4

This is just a guess. I don't have any evidence supporting the following claims at the moment but my guess is that this is a simple overflow problem: 

>>> np.arange(2 ** 31 - 1).size
2147483647

Which just happens to be the largest int32 value:

>>> np.iinfo(np.int32)
iinfo(min=-2147483648, max=2147483647, dtype=int32)

So when you actually have an array with a size of 2147483648 (2**31) and use an int32 this would overflow and give an actual negative value. Then there is probably something like this inside the numpy.ndarray.copy method:

for (i = 0 ; i < size ; i ++) {
    newarray[i] = oldarray[i]
}

But given that the size is now negative the loop wouldn't execute because 0 > -2147483648.

That the new array is actually initialized with zeros is strange because it wouldn't make sense to actually put zeros before one copies the array (but it could be something like in this question).

Again: That's just guessing at this point but it would match the behaviour.

Community
  • 1
  • 1
MSeifert
  • 145,886
  • 38
  • 333
  • 352