python3/numpy: in place addition inconsistency when adding a complex number

Question

I was testing a piece of code and encountered the following behaviour. Although I have found an explanation and a solution myself (this looks related although not same), I'd like to ask here if anybody of you has more to add.

a) I cannot in-place-add a complex number to a numpy array that was created as real, but I can do it with a scalar variable

b) warning messages are issued only in a limited number of cases, and not very consistently

Take a look at these commands:

>>> from numpy import *
>>> a5=arange(5)
>>> a5[1:5]+=1j
>>> a5
array([0, 1, 2, 3, 4])
>>> a=8
>>> a+=1j
>>> a
(8+1j)
>>> a5+=1j
__main__:1: ComplexWarning: Casting complex values to real discards the imaginary part
>>> a5[3]+=1j
>>> a5[3]
3
>>> 

You see? A ComplexWarning was issued only when trying to add a complex to the whole vector, but not when attempting the same in place addition to a single element or a slice.

Is there a reason for this? (I barely can stand the fact that it doesn't make the addition, but certainly not the fact that the warning is issued only in one case). Is this a bug or a feature? How can it be best overcome? At least, wouldn't it be better that the ComplexWarning appear in all cases when the imaginary part is discarded for incompatibility?

PS. Meanwhile I provide a solution: z5=arange(5)+0j and c5=arange(5,dtype="complex"), will both behave as intended. But the first is is only partial relief, as it shows that a numpy range of integers CAN be added and assigned to a variable, provided that it is a complex one, or a new one being created.

Answer 1

This is recognised as an issue, but it has taken time to change as fixing it breaks compatibility.

Since numpy 1.10, the default casting rule for in-place operations and for ufuncs has been changed to "same_kind", so your code will issue a TypeError:

TypeError: Cannot cast ufunc add output from dtype('complex128') to dtype('int64') with casting rule 'same_kind'

From the release notes:

Default casting for inplace operations will change to ‘same_kind’ in Numpy 1.10.0. This will certainly break some code that is currently ignoring the warning.

Answer 2

The warning is issued every time by NumPy, but Python's general warnings machinery by default only prints the warning the first time it's issued. You can change this behaviour using warnings.filterwarnings or warnings.simplefilter. See the warnings module documentation for more information.

Here's an example (with Python 2.7 and NumPy 1.9), showing that the warning is being issued for each one of the in-place operations.

Python 2.7.10 |Master 2.1.0.dev1829 (64-bit)| (default, Oct 21 2015, 09:09:19) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.6)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from numpy import *
>>> import warnings
>>> warnings.simplefilter('always', ComplexWarning)
>>> a5 = arange(5)
>>> a5[1:5] += 1j
__main__:1: ComplexWarning: Casting complex values to real discards the imaginary part
>>> a5
array([0, 1, 2, 3, 4])
>>> a5 += 1j
__main__:1: ComplexWarning: Casting complex values to real discards the imaginary part
>>> a5[3] += 1j
__main__:1: ComplexWarning: Casting complex values to real discards the imaginary part

The scalar addition that you show is a different beast altogether.

>>> a = 8
>>> a += 1j

In this case, the += operation is not doing an in-place operation (it can't, since after the first line, a is a Python int, and unlike NumPy arrays, int objects are immutable). Instead, it's equivalent to doing a = a + 1j. That's constructing a new Python complex object, and rebinding a to refer to that new object.

You say you "barely can stand the fact that it doesn't make the addition". But note that there's no way to efficiently do so: the contents of the original array are stored using 8-bytes per item. The contents of the sum are (double-precision) complex numbers, so would have to be stored using 16 bytes per item. That makes a true in-place operation impossible here.

Note that in NumPy 1.10, these operations will produce a TypeError rather than a warning.