View of numpy structured array with offsets

Question

I have the following numpy structured array:

In [250]: x
Out[250]: 
array([(22, 2, -1000000000, 2000), (22, 2, 400, 2000),
       (22, 2, 804846, 2000), (44, 2, 800, 4000), (55, 5, 900, 5000),
       (55, 5, 1000, 5000), (55, 5, 8900, 5000), (55, 5, 11400, 5000),
       (33, 3, 14500, 3000), (33, 3, 40550, 3000), (33, 3, 40990, 3000),
       (33, 3, 44400, 3000)], 
      dtype=[('f1', '<i4'), ('f2', '<i4'), ('f3', '<i4'), ('f4', '<i4')])

The array below is a subset (also a view) of the above array:

In [251]: fields=['f1','f3']

In [252]: y=x.getfield(np.dtype(
     ...:                    {name: x.dtype.fields[name] for name in fields}
     ...:                    ))

In [253]: y
Out[253]: 
array([(22, -1000000000), (22, 400), (22, 804846), (44, 800), (55, 900),
       (55, 1000), (55, 8900), (55, 11400), (33, 14500), (33, 40550),
       (33, 40990), (33, 44400)], 
      dtype={'names':['f1','f3'], 'formats':['<i4','<i4'], 'offsets':[0,8], 'itemsize':12})

I am trying to convert y to a regular numpy array. I want the array to be a view. The issue is that the following gives me an error:

In [254]: y.view(('<i4',2))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-254-88440f106a89> in <module>()
----> 1 y.view(('<i4',2))

C:\numpy\core\_internal.pyc in _view_is_safe(oldtype, newtype)
    499 
    500     # raises if there is a problem
--> 501     _check_field_overlap(new_fieldtile, old_fieldtile)
    502 
    503 # Given a string containing a PEP 3118 format specifier,

C:\numpy\core\_internal.pyc in _check_field_overlap(new_fields, old_fields)
    402         old_bytes.update(set(range(off, off+tp.itemsize)))
    403     if new_bytes.difference(old_bytes):
--> 404         raise TypeError("view would access data parent array doesn't own")
    405 
    406     #next check that we do not interpret non-Objects as Objects, and vv

TypeError: view would access data parent array doesn't own

However, if i choose consecutive fields it works:

In [255]: fields=['f1','f2']
     ...: 
     ...: y=x.getfield(np.dtype(
     ...:                    {name: x.dtype.fields[name] for name in fields}
     ...:                    ))
     ...: 

In [256]: y
Out[256]: 
array([(22, 2), (22, 2), (22, 2), (44, 2), (55, 5), (55, 5), (55, 5),
       (55, 5), (33, 3), (33, 3), (33, 3), (33, 3)], 
      dtype=[('f1', '<i4'), ('f2', '<i4')])

In [257]: y.view(('<i4',2))
Out[257]: 
array([[22,  2],
       [22,  2],
       [22,  2],
       [44,  2],
       [55,  5],
       [55,  5],
       [55,  5],
       [55,  5],
       [33,  3],
       [33,  3],
       [33,  3],
       [33,  3]])

View casting seems to not work when the fields are not contiguous, is there an alternative?

Answer 1

Yes, use the ndarray constructor directly:

x = np.array([(22, 2, -1000000000, 2000), 
              (22, 2,         400, 2000),
              (22, 2,      804846, 2000), 
              (44, 2,         800, 4000), 
              (55, 5,         900, 5000), 
              (55, 5,        1000, 5000)], 
             dtype=[('f1','i'),('f2','i'),('f3','i'),('f4','i')])

fields = ['f4', 'f1']
shape = x.shape + (len(fields),)
offsets = [x.dtype.fields[name][1] for name in fields]
assert not any(np.diff(offsets, n=2))
strides = x.strides + (offsets[1] - offsets[0],)
y = np.ndarray(shape=shape, dtype='i', buffer=x,
               offset=offsets[0], strides=strides)
print repr(y)

Gives:

array([[2000,   22],
       [2000,   22],
       [2000,   22],
       [4000,   44],
       [5000,   55],
       [5000,   55]])

---

Btw, when all fields in the original array have the same dtype, it's much easier to first create a view on that array followed by a slicing operation. For the same result as above:

y = x.view('i').reshape(x.shape + (-1,))[:,-1::-3]

Answer 2

The following is a bit confusing - but the gist is that for this kind of view to work it has to be able to access the fields with regular array strides and shapes. Getting a view from ['f1','f3'] fails for basically the same reason that np.ones((12,4))[:,[0,2]] produces a copy.

========

In your structured array, each record is stored as 4*'i4' bytes. That layout is compatible with a (n,4) 'i4' array:

In [381]: x.__array_interface__['data']    # databuffer pointer
Out[381]: (160925352, False)
In [382]: x.view(('i',4)).__array_interface__['data']
Out[382]: (160925352, False)          # same buffer
In [387]: x.view(('i',4)).shape
Out[387]: (12, 4)

but when I take various slices of this array

In [383]: x.view(('i',4))[:,[0,1]].__array_interface__['data']
Out[383]: (169894184, False)       # advance indexing - a copy

In [384]: x.view(('i',4))[:,:2].__array_interface__['data']
Out[384]: (160925352, False)       # same buffer

But selecting ['f1','f3'] is equivalent to: x.view(('i',4))[:,[0,2]], another copy.

Or look at the strides. With the 1st 2 fields

In [404]: y2=x.getfield(np.dtype({name: x.dtype.fields[name] for name in ['f1','f2']}))
In [405]: y2.dtype
Out[405]: dtype([('f1', '<i4'), ('f2', '<i4')])
In [406]: y2.strides
Out[406]: (16,)
In [407]: y2.view(('i',2)).strides
Out[407]: (16, 4)

To see this array as just ints, it can step by 16 for rows, and by 4 to step columns and just take 2 columns.

Or look at the full dictionary for the 4 column and 2 column cases

In [409]: x.view(('i',4)).__array_interface__
Out[409]: 
{'data': (160925352, False),
 'descr': [('', '<i4')],
 'shape': (12, 4),
 'strides': None,
 'typestr': '<i4',
 'version': 3}
In [410]: y2.view(('i',2)).__array_interface__
Out[410]: 
{'data': (160925352, False),
 'descr': [('', '<i4')],
 'shape': (12, 2),
 'strides': (16, 4),
 'typestr': '<i4',
 'version': 3}

same strides and dtype, just different shape. The y2 case works because it can access the desired bytes with striding, and ignoring 2 columns.

If I slice out 2 middle columns of the 4 column case, I get a view - same data buffer, but with an offset:

In [385]: x.view(('i',4))[:,2:4].__array_interface__['data']
Out[385]: (160925360, False)

but using getfield with those 2 fields gives the same error as with ['f1','f3']:

In [388]: y2=x.getfield(np.dtype({name: x.dtype.fields[name] for name in ['f2','f3']})).view(('i',2))
...
ValueError: new type not compatible with array.

view can't implement that databuffer offset that slicing can.

========

Looking again at the 2 middle fields:

In [412]: y2=x.getfield(np.dtype({name: x.dtype.fields[name] for name in ['f2','f3']}))
     ...:  
In [413]: y2
Out[413]: 
array([(2, -1000000000), (2, 400), (2, 804846), (2, 800), (5, 900),
       (5, 1000), (5, 8900), (5, 11400), (3, 14500), (3, 40550),
       (3, 40990), (3, 44400)], 
      dtype={'names':['f2','f3'], 'formats':['<i4','<i4'], 'offsets':[4,8], 'itemsize':12})
In [414]: y2.__array_interface__['data']
Out[414]: (160925352, False)

y2 is pointing at the original database start. It achieves the offset with the dtype offsets. Compare that with the offset in In[385].