0

I have a process that requires extracting data from a numpy recarray to a ndarray where I then do some vector math. (The recarray comes from a pytables table.read() function.) I want to map the math output (another ndarray) back to the same fields/columns in the original recarray. I found a way to do it column by column. Looking for a better way to go back and forth with the data. my code:

node_eigen_array = eigenvb_table.read_coordinates(node_rows)
node_eigen_array.shape[0]
10
node_eigen_array.dtype
dtype([('ID', '<i8'), ('X', '<f8'), ('Y', '<f8'), ('Z', '<f8'),  ('FREQ', '<i8')])
resvec_array[:,0]=node_eigen_array['X']
resvec_array[:,1]=node_eigen_array['Y']
resvec_array[:,2]=node_eigen_array['Z']

# do some stuff that returns ndarray c_dot...

node_eigen_array['X']=cdot[:,0]
node_eigen_array['Y']=cdot[:,1]
node_eigen_array['Z']=cdot[:,2]

I tried this to skip the first recarray to ndarray: 

resvec_array=node_eigen_array[['X','Y','Z']].view('float64').reshape((10,3))

numpy complains:

This code may break in numpy 1.13 because this will return a view instead of a copy -- see release notes for details.

Also, looking for something that simplifies ndarray data back to recarray. Thanks.

kcw78
  • 7,131
  • 3
  • 12
  • 44

1 Answers1

1

This is a future warning, not an error. The change has been postponed to 1.16. It has to do with multifield indexing, your [['X','Y','Z']] step.

In [56]: dt = np.dtype([('ID', '<i8'), ('X', '<f8'), ('Y', '<f8'), ('Z', '<f8'), ('FREQ', '<i8')])
In [57]: arr = np.ones(3, dtype=dt)
In [58]: arr       # a structured array, recarray is just variation
Out[58]: 
array([(1, 1., 1., 1., 1), (1, 1., 1., 1., 1), (1, 1., 1., 1., 1)],
      dtype=[('ID', '<i8'), ('X', '<f8'), ('Y', '<f8'), ('Z', '<f8'), ('FREQ', '<i8')])

It's quiet when you just view the fields:

In [59]: arr[['X','Y','Z']]
Out[59]: 
array([(1., 1., 1.), (1., 1., 1.), (1., 1., 1.)],
      dtype=[('X', '<f8'), ('Y', '<f8'), ('Z', '<f8')])

But it warns of a change when you try to do something with them. Note it still does the action.

In [60]: arr[['X','Y','Z']].view('float64')
/usr/local/bin/ipython3:1: FutureWarning: Numpy has detected that you may be viewing or writing to an array returned by selecting multiple fields in a structured array. 

This code may break in numpy 1.16 because this will return a view instead of a copy -- see release notes for details.
  #!/usr/bin/python3
Out[60]: array([1., 1., 1., 1., 1., 1., 1., 1., 1.])

A way to silence the warning is to add copy() after the indexing:

In [62]: arr[['X','Y','Z']].copy().view('float64')
Out[62]: array([1., 1., 1., 1., 1., 1., 1., 1., 1.])

Currently this view change works. But in the planned change, the arr[['X','Y','Z']] data layout will be different, and the view won't work. There's some complex business about offsets.

hpaulj
  • 221,503
  • 14
  • 230
  • 353
  • Thanks. Yeah, originally I had the .copy()...think I overlooked that this morning. So, any tips to copy values from ndarray c_dot back to matching arr[['ID', 'X','Y','Z', 'FREQ']] layout? – kcw78 Oct 05 '18 at 20:03
  • I wouldn't worry about doing field by field copies, in either direction. You have 3 fields, and how many records - 1000s? It's standard proceedure in `recfunctions` to do field by field copies. – hpaulj Oct 05 '18 at 20:37
  • I could easily have 10e3 to 100e3 or even 1e6 records, but performance wasn't my first concern. I was looking for code simplicity, as I have to do this with other entries. That said, I have a working procedure, and it sounds like it's a good one. I just need to be careful with the indices each time. Thanks for your insights!! – kcw78 Oct 05 '18 at 21:07
  • The newest recfunctions has structured to/from unstructured functions. See 1.16 docs – hpaulj Mar 21 '19 at 18:06