Unexpected result from boolean mask slicing

Question

I'm confused about the way numpy array slicing is working in the example below. I can't figure out how exactly the slicing is working and would appreciate an explanation.

import numpy as np
arr = np.array([
    [1,2,3,4],
    [5,6,7,8],
    [9,10,11,12],
    [13,14,15,16]
    ])
m = [False,True,True,False]

# Test 1 - Expected behaviour
print(arr[m])
Out: 
array([[ 5,  6,  7,  8],
       [ 9, 10, 11, 12]])

# Test 2 - Expected behaviour
print(arr[m,:])
Out:
array([[ 5,  6,  7,  8],
       [ 9, 10, 11, 12]])

# Test 3 - Expected behaviour
print(arr[:,m])
Out:
array([[ 2,  3],
       [ 6,  7],
       [10, 11],
       [14, 15]])

### What's going on here? ###
# Test 4
print(arr[m,m])
Out:
array([ 6, 11]) # <--- diagonal components. I expected [[6,7],[10,11]].

I found that I could achieve the desired result with arr[:,m][m]. But I'm still curious about how this works.

Docs should help - https://numpy.org/devdocs/user/basics.indexing.html#indexing-multi-dimensional-arrays, https://numpy.org/doc/1.18/reference/arrays.indexing.html#purely-integer-array-indexing — Divakar, Jun 05 '20 at 13:04
It seems `m = [False, True, True, False]` is treated as if it is `m = [1,2]` indicating the positions of `True`. Then it computes `arr[[1,2],[1,2]]`. This doesn't seem sensible. The latter syntax can be (and probably is, mostly) used to reorder to reorder the rows and columns of a matrix while selecting elements. But the former syntax with `bool` forces a particular order. It doesn't have the same power, so why give it this unintuitive behaviour? — Myridium, Jun 05 '20 at 13:30
So, what are you expecting with `arr[m,m]` to give out and why? — Divakar, Jun 05 '20 at 13:33
With `arr[n, m]` where `n` and `m` are arrays (or lists), `numpy` `broadcasts` the arrays against each other to create a nd selection array. The indexing documentation should be clear on this - take time to read and digest https://numpy.org/doc/1.18/reference/arrays.indexing.html (yes I know it is longer than a SO sound bite). — hpaulj, Jun 05 '20 at 16:49
@Divakar - I expected `arr[n,m]` to broadcast `n` as if it were a column vector, and broadcast `m` as if it were a row vector, and to apply the logical conjunction of these selection masks. E.g. the intersection of the rows indicated by `n` and the columns indicated by `m`. — Myridium, Jun 06 '20 at 02:09
@hpaulj - the reference is poorly written. For example _"If obj.ndim == x.ndim, x[obj] returns a 1-dimensional array filled with the elements of x corresponding to the True values of obj"_ - this information is false as written, as it implicitly assumes that `obj` has a compatible shape with `x`, and that if it doesn't, one is to assume that `obj` is broadcasted to match `x`. It also says _"Advanced indexing always returns a copy of the data (contrast with basic slicing that returns a view)."_ and yet when I perform assignment on this ""copy"" I alter the original object (confirmed with `id`). — Myridium, Jun 06 '20 at 02:30

LevB · Answer 1 · 2020-06-06T21:41:08.157

1

You can use matrix multiplication to create a 2d mask.

import numpy as np
arr = np.array([
[1,2,3,4],
[5,6,7,8],
[9,10,11,12],
[13,14,15,16]
])
m = [False,True,True,False]

mask2d = np.array([m]).T * m
print(arr[mask2d])

Output :

[ 6  7 10 11]

Alternatively, you can have the output in matrix format.

print(np.ma.masked_array(arr, ~mask2d))

edited Jun 06 '20 at 21:41

answered Jun 06 '20 at 21:22

LevB

925
6
10

score 0 · Answer 2 · answered Jun 05 '20 at 13:45

It's just the way indexing works for numpy arrays. Usually if you have specific "slices" of rows and columns you want to select you just do:

import numpy as np
arr = np.array([
    [1,2,3,4],
    [5,6,7,8],
    [9,10,11,12],
    [13,14,15,16]
    ])

# You want to check out rows 2-3 cols 2-3
print(arr[2:4,2:4])
Out:
[[11 12]
 [15 16]]

Now say you want to select arbitrary combinations of specific row and column indices, for example you want row0-col2 and row2-col3

print(arr[[0, 2], [2, 3]])
Out:
[ 3 12]

What you are doing is identical to the above. [m,m] is equivalent to:

[m,m] == [[False,True,True,False], [False,True,True,False]]

Which is in turn equivalent to saying you want row1-col1 and row2-col2

print(arr[[1, 2], [1, 2]])
Out:
[ 6 11]

Myridium · Accepted Answer · 2020-06-07T23:14:26.757

0

I don't know why, but this is the way numpy treats slicing by a tuple of 1d boolean arrays:

arr = np.array([
    [1,2,3,4],
    [5,6,7,8],
    [9,10,11,12]
    ])
m1 = [True, False, True]
m2 = [False, False, True, True]

# Pseudocode for what NumPy does
#def arr[m1,m2]:
#  intm1 = np.transpose(np.argwhere(m1)) # [True, False, True] -> [0,2]
#  intm2 = np.transpose(np.argwhere(m2)) # [False, False, True, True] -> [2,3]
#  return arr[intm1,intm2] # arr[[0,2],[2,3]]

print(arr[m1,m2]) # --> [3 12]

What I was expecting was slicing behaviour with non-contiguous segments of the array; selecting the intersection of rows and columns, can be achieved with:

arr = np.array([
    [1,2,3,4],
    [5,6,7,8],
    [9,10,11,12]
    ])
m1 = [True, False, True]
m2 = [False, False, True, True]

def row_col_select(arr, *ms): 
   n = arr.ndim 
   assert(len(ms) == n) 
   # Accumulate a full boolean mask which will have the shape of `arr`
   accum_mask = np.reshape(True, (1,) * n) 
   for i in range(n): 
     shape = tuple([1]*i + [arr.shape[i]] + [1]*(n-i-1)) 
     m = np.reshape(ms[i], shape) 
     accum_mask = np.logical_and(accum_mask, m) 
   # Select `arr` according to full boolean mask
   # The boolean mask is the multiplication of the boolean arrays across each corresponding dimension. E.g. for m1 and m2 above it is:
   # m1:   | m2: False False  True  True
   #       |       
   # True  |   [[False False  True  True]
   # False |    [False False False False]
   # True  |    [False False  True  True]]
   return arr[accum_mask]

print(row_col_select(arr,m1,m2)) # --> [ 3  4 11 12]

edited Jun 07 '20 at 23:14

answered Jun 06 '20 at 03:08

Myridium

789
10
20

Do you expect this block behavior for index lists and arrays as well? Or just booleans? What would be the syntax for picking elements across rows such as [3, 12]? – hpaulj Jun 07 '20 at 14:55
@hpaulj - just booleans. That would be `arr[[False,False,False,True,True, True, True, True, True, True, True, True, True],m2]` where `m2` determines the columns. – Myridium Jun 07 '20 at 23:02
@hpaulj - the implication with a boolean value is that there are two options-- included or not included. My interpretation is a natural one. Is the row included or is it not? Is the column included or is it not? Simple. The way it works at the moment, you need exactly the same number of bools to be `True` in each 1d mask. Then it assumes a particular order on the elements which you have no control over... I'm sorry but it's just nonsensical. I can always turn the boolean array into an integer array if I want to. In addition, the interpretation I have provided has additional utility.... – Myridium Jun 07 '20 at 23:17
...as it extends the existing 'view' capabilities of numpy arrays to also view *non-contiguous blocks*. Actually, if there isn't any inbuilt way to do this already then I must say I'm surprised. We aren't always going to be slicing contiguous segments. I thought to myself "How would we slice non-contiguous segments? Presumably numpy can do this out of the box." and this is the syntax I came up with. – Myridium Jun 07 '20 at 23:20
You can get a non-contiguous view by using slices like `arr[::2, ::3]`. The accesses the same data buffer, but with different strides and shape. That isn't possible with the more general selection via arrays (integer or boolean). – hpaulj Jun 07 '20 at 23:59
@hpaulj - that's a very specific use case of every second or every third element... – Myridium Jun 08 '20 at 04:24
I showed you how to index noncontiguous blocks, in [63] or with `ix_`. But you have to use integer arrays, not booleans. The fact that integer indexes can reorder the rows doesn't make a difference. If the selection can't be expressed as strides, it has to be a copy. – hpaulj Jun 08 '20 at 16:05
Yes, thank you for showing me the `ix_`. The statement `(arr[np.ix_(m1,m2)]) = -1` alters the original. I have put in explicit parentheses to ensure nothing funky is going on with the `__equals__`. It modifies the original. Therefore it does not return a copy of the data, unless perhaps the copy has had the assignment operator overwritten to alter the parent object by some stored link inside the copy. – Myridium Jun 09 '20 at 07:35
When you use `advanced indexing` in an assignment, the distinction between `view` and `copy` does not apply. With assignment, the indexing tuple is passed directly to the `__setitem__` method/ But in `arr[idx1][idx2] = value`, there's an intermediate `__getitem__` call that must produce a `view`. – hpaulj Jun 09 '20 at 16:31

hpaulj · Answer 4 · 2020-06-07T23:50:09.737

In [55]: arr = np.array([ 
    ...:     [1,2,3,4], 
    ...:     [5,6,7,8], 
    ...:     [9,10,11,12], 
    ...:     [13,14,15,16] 
    ...:     ]) 
    ...: m = [False,True,True,False]

In all your examples we can use this m1 instead of the boolean list:

In [58]: m1 = np.where(m)[0]                                                    
In [59]: m1                                                                     
Out[59]: array([1, 2])

If m was a 2d array like arr than we could use it to select elements from arr - but they will be raveled; but when used to select along one dimension, the equivalent array index is clearer. Yes we could use np.array([2,1]) or np.array([2,1,1,2]) to select rows in a different order or even multiple times. But substituting m1 for m does not loose any information or control.

Select rows, or columns:

In [60]: arr[m1]                                                                
Out[60]: 
array([[ 5,  6,  7,  8],
       [ 9, 10, 11, 12]])
In [61]: arr[:,m1]                                                              
Out[61]: 
array([[ 2,  3],
       [ 6,  7],
       [10, 11],
       [14, 15]])

With 2 arrays, we get 2 elements, arr[1,1] and arr[2,2].

In [62]: arr[m1, m1]                                                            
Out[62]: array([ 6, 11])

Note that in MATLAB we have to use sub2ind to do the same thing. What's easy in numpy is a bit harder in MATLAB; for blocks it's the other way.

To get a block, we have to create a column array to broadcast with the row one:

In [63]: arr[m1[:,None], m1]                                                    
Out[63]: 
array([[ 6,  7],
       [10, 11]])

If that's too hard to remember, np.ix_ can do it for us:

In [64]: np.ix_(m1,m1)                                                          
Out[64]: 
(array([[1],
        [2]]),
 array([[1, 2]]))

[63] is doing the same thing as [62]; the difference is that the 2 arrays broadcast differently. It's the same broadcasting as done in these additions:

In [65]: m1+m1                                                                  
Out[65]: array([2, 4])
In [66]: m1[:,None]+m1                                                          
Out[66]: 
array([[2, 3],
       [3, 4]])

This indexing behavior is perfectly consistent - provided we don't import expectations from other languages.

I used m1 because boolean arrays don't broadcast, as show below:

In [67]: np.array(m)                                                            
Out[67]: array([False,  True,  True, False])
In [68]: np.array(m)[:,None]                                                    
Out[68]: 
array([[False],
       [ True],
       [ True],
       [False]])
In [69]: arr[np.array(m)[:,None], np.array(m)]                                  
...
IndexError: too many indices for array

in fact the 'column' boolean doesn't work either:

In [70]: arr[np.array(m)[:,None]]                                               
...
IndexError: boolean index did not match indexed array along dimension 1; dimension is 4 but corresponding boolean dimension is 1

We can use logical_and to broadcast a column boolean against a row boolean:

In [72]: mb = np.array(m)                                                       
In [73]: mb[:,None]&mb                                                          
Out[73]: 
array([[False, False, False, False],
       [False,  True,  True, False],
       [False,  True,  True, False],
       [False, False, False, False]])
In [74]: arr[_]                                                                 
Out[74]: array([ 6,  7, 10, 11])      # 1d result

This is the case you quoted: "If obj.ndim == x.ndim, x[obj] returns a 1-dimensional array filled with the elements of x corresponding to the True values of obj"

Your other quote:

*"Advanced indexing always returns a copy of the data (contrast with basic slicing that returns a view)." *

means that if arr1 = arr[m,:], arr1 is a copy, and any modifications to arr1 will not affect arr. However I could use arr[m,:]=10to modify arr. The alternative to a copy is a view, as in basic indexing, arr2=arr[0::2,:]. modifications to arr2 do modify arr as well.

Unexpected result from boolean mask slicing

4 Answers4