How to add or construct nested numpy structured array from existing structures

Question

In general, my question is on the possible ways for creating/appending to nested structured arrays. Specifically where the dtype structure is known, but the size in the nested elements weren't pre-defined. I have tried different things with the join_by and fromarray methods, but haven't had success. The doc examples seem to suggest the join type methods are useful for a sort of 'zipping' of data and not really extending it?

Datatypes defined

from numpy.lib import recfunctions as rfn
c = [('x','f8'),('y','f8')]
d = [('text','|S20'),('val', c)]

Creates 2 rows of 'd'

zeros = np.zeros(2, dtype=d)
print(f'{zeros=}')

How to make multiple rows of 'c'?

c = [('x','f8'),('y','f8')]
# Add a number to the construct for 'c'
d = [('text','|S20'),('val', c, 3)]
zeros = np.zeros(2, dtype=d)
print(f'{zeros=}')

How to do this without predefining the size of 'c'? Appending/Extending nested elements doesn't seem to work At the lowest level, 'c' can be used to construct a recarray

large_c = np.arange(2*3).reshape(3,2)
struct_c = rfn.unstructured_to_structured(large_c, np.dtype(c))
print(f'{struct_c=}')

But now, is there a way to construct the next level from existing structured arrays? I thought constructing an array with a shape that seems to broadcast would work, but it fails

struct_d = rfn.unstructured_to_structured(np.array(['', large_c]), np.dtype(d))
print(f'{struct_d=}')
>> The length of the last dimension of arr must be equal to the number of fields in dtype

I also tried the recarray constructions, which had their own issues

struct_d = np.core.records.fromarrays(np.array(['', large_c]), dtype=d)
print(f'{struct_d=}')
>>> array-shape mismatch in array 1

That `unstructured_to_structured` (and its reverse) is about the only function in `recfunctions` I use/recommend these days. I have an idea of what the other functions are doing, and have answered a few questions, but for the most part I prefer to construct the dtype myself, and assign values by-field. Nested dtypes, while fun, don't seem to have a lot of practical use. — hpaulj, Jan 12 '22 at 20:45
@hpaulj It is fun for sure, but also feels like it *could* be a nice elegant way to format and work with certain data (in particular, data that is already organized like this in a corresponding C dll). Perhaps im learning the limits pretty quickly. But, it feels like its possible to properly broadcast this. I am able to totally flatten `large_c` and insert an empty string at the first index, which then broadcasts to the nested format, but this is much messier to track/test. — 001001, Jan 12 '22 at 21:57

score 0 · Answer 1 · answered Jan 13 '22 at 00:48

I think the most straightforward way of making such an array is:

In [352]: c = [('x','f8'),('y','f8')]
     ...: # Add a number to the construct for 'c'
     ...: d = [('text','|S20'),('val', c, 3)]
     ...: zeros = np.zeros(2, dtype=d)
In [353]: zeros
Out[353]: 
array([(b'', [(0., 0.), (0., 0.), (0., 0.)]),
       (b'', [(0., 0.), (0., 0.), (0., 0.)])],
      dtype=[('text', 'S20'), ('val', [('x', '<f8'), ('y', '<f8')], (3,))])

In [355]: x = np.arange(12).reshape(2,3,2)

In [357]: carr = rf.unstructured_to_structured(x,np.dtype(c))
In [358]: carr
Out[358]: 
array([[( 0.,  1.), ( 2.,  3.), ( 4.,  5.)],
       [( 6.,  7.), ( 8.,  9.), (10., 11.)]],
      dtype=[('x', '<f8'), ('y', '<f8')])

With the right shape and dtype, it can be assigned to the larger array:

In [359]: zeros['val']=carr
In [360]: zeros
Out[360]: 
array([(b'', [( 0.,  1.), ( 2.,  3.), ( 4.,  5.)]),
       (b'', [( 6.,  7.), ( 8.,  9.), (10., 11.)])],
      dtype=[('text', 'S20'), ('val', [('x', '<f8'), ('y', '<f8')], (3,))])

I'm not sure if there's something in your question that I glossed over or not.

How to add or construct nested numpy structured array from existing structures

1 Answers1