1

I have a np.ndarray (call it arr) that looks something like this:

# python 3.7
import numpy as np

my_dtype = [("x", "float32"), ("y", "float32"),
                ("some_more", "int32"), ("and_more_stuff", "uint8")]  #
part1 = np.zeros(5, dtype=my_dtype)  # various lengths (here e.g. 5, 6 and 7)
part2 = np.zeros(6, dtype=my_dtype)
part3 = np.zeros(7, dtype=my_dtype)
# ... an a priori unknown number of "parts". Also of course there are values inside, not just zeros.

arr = np.array([part1, part2, part3])

(maybe I should not use numpy arrays to handle such data?)

Now I want to do things with arr. For example I want to find the total minimum among all values "x" and "y" in all subarrays (a single number). My solution looks fairly horrible which means that I don't understand how to use structured arrays (despite reading the documentation and the official tutorial):

arr[0][["x", "y"]][1] = (-3., 4.5) # put in some values as an example
all_x_y_coords= np.array([[list(mytuple) for mytuple in mylist] for mylist in
                   [list(part[["x", "y"]]) for part in arr]])
print(np.min(np.min(all_x_y_coords))) # gives -3. as desired, but at what cost?!

Doing something like this is clearly not practical. How would one calculate the minimum that I want? The next thing I would like to do is apply a rotation matrix to all "x,y". Before I concoct something even more horrible than the code above I thought I had better understand what I'm doing wrong. Thank you very much in advance for any help!

Adomas Baliuka
  • 1,384
  • 2
  • 14
  • 29
  • 2
    Why are you combining the 3 arrays into one - why not a list? Doing 'math' across fields of a structured array isn't easy. You need to treat the two fields as separate arrays. – hpaulj Nov 28 '19 at 18:54
  • 1
    you might want to look into using pandas, its good for structured arrays and for instance allows different types for different columns – jeremy_rutman Nov 28 '19 at 19:35
  • @hpaulj What advantages would a list have in this a case? I like to be able to easily save the entire arr in one .npy file. If I had a list of partx arrays, how would it become easier to perform operations, such as the ninimum and rotation that I want? – Adomas Baliuka Nov 29 '19 at 12:56
  • `for part in alist` is faster than `for part in arr` – hpaulj Nov 29 '19 at 13:23

2 Answers2

2
In [167]: part1[['x','y']][1]=(-3, 4.5)                                         
In [168]: part1                                                                 
Out[168]: 
array([( 0., 0. , 0, 0), (-3., 4.5, 0, 0), ( 0., 0. , 0, 0),
       ( 0., 0. , 0, 0), ( 0., 0. , 0, 0)],
      dtype=[('x', '<f4'), ('y', '<f4'), ('some_more', '<i4'), ('and_more_stuff', 'u1')])

Since they all have the same dtype, they can be joined into one array:

In [169]: arr = np.concatenate([part1,part2,part3])                             
In [170]: arr.shape                                                             
Out[170]: (18,)
In [171]: arr.dtype                                                             
Out[171]: dtype([('x', '<f4'), ('y', '<f4'), ('some_more', '<i4'), ('and_more_stuff', 'u1')])
In [172]: arr['x']                                                              
Out[172]: 
array([ 0., -3.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
        0.,  0.,  0.,  0.,  0.], dtype=float32)
In [173]: np.min(arr['x'])                                                      
Out[173]: -3.0
In [174]: np.min(arr['y'])                                                      
Out[174]: 0.0

Joining them with np.array just makes an object dtype array, little better (and argueablly worse) than a list:

In [175]: arr1 = np.array([part1,part2,part3])                                  
In [176]: arr1.shape                                                            
Out[176]: (3,)
In [177]: arr1.dtype                                                            
Out[177]: dtype('O')

There's little we can do with such an array without some sort of explicit iteration over the 3 elements.

hpaulj
  • 221,503
  • 14
  • 230
  • 353
  • The issue is that it is crucial to keep track which data belongs to which part and the parts have different sizes. I don't see a good way to keep that information using `concatenate` – Adomas Baliuka Nov 29 '19 at 12:59
2

it is possible, without getting into the 'recarray' option. I work with structured arrays all the time to represent geometry objects coming from a variety of sources.

from numpy.lib.recfunctions import structured_to_unstructured as stu
pnts
array([( 0, 10. , 10. , 4), ( 1, 10. ,  0. , 2), ( 2,  1.5,  1.5, 1),
       ( 3,  0. , 10. , 1), ( 5,  3. ,  9. , 2), ( 6,  3. ,  3. , 1),
       ( 7,  9. ,  3. , 1), ( 8,  9. ,  9. , 1), (10,  2. ,  7. , 2),
       (11,  1. ,  7. , 1), (12,  2. ,  5. , 1), (14,  2. ,  8. , 2),
       (15,  1. ,  9. , 1), (16,  1. ,  8. , 1), (18,  8. ,  8. , 2),
       (19,  8. ,  4. , 1), (20,  4. ,  4. , 1), (21,  5. ,  7. , 1),
       (23,  6. ,  7. , 2), (24,  5. ,  5. , 1), (25,  7. ,  5. , 1),
       (27, 25. , 14. , 2), (28, 25. ,  4. , 1), (29, 15. ,  4. , 1),
       (30, 15. ,  6. , 1), (31, 23. ,  6. , 1), (32, 23. , 12. , 1),
       (33, 15. , 12. , 1), (34, 15. , 14. , 1), (36, 20. , 10. , 2),
       (37, 20. ,  8. , 1), (38, 12. ,  8. , 1), (39, 12. ,  2. , 1),
       (40, 20. ,  2. , 1), (41, 20. ,  0. , 1), (44, 14. , 10. , 3),
       (46, 11. ,  9. , 2), (47, 12. ,  8.5, 1), (48, 12. ,  9. , 1),
       (50, 10.5,  8.5, 2), (51, 10.5,  7. , 1), (52, 11.5,  7. , 1),
       (54, 10.5,  2. , 2), (55, 10.5,  0.5, 1), (56, 11.5,  0.5, 1),
       (60, 15. , 18. , 1)],
      dtype=[('New_ID', '<i4'), ('Xs', '<f8'), ('Ys', '<f8'), ('Num', '<i4')])

np.mean(stu(pnts[['Xs', 'Ys']]),axis=0)     # --- array([10.58,  6.77])
# or
(np.mean(pnts['Xs']), np.mean(pnts['Ys']))  # --- (10.576086956521738, 6.771739130434782)

Option 2... keep the data structure, then convert it when appropriate

pnts2 = stu(pnts[['Xs', 'Ys']])

pnts2
array([[10. , 10. ],
       [10. ,  0. ],
       [ 1.5,  1.5],
... snip
       [10.5,  0.5],
       [11.5,  0.5],
       [15. , 18. ]])

np.mean(pnts2, axis=0)  # ---- array([10.58,  6.77])
NaN
  • 2,212
  • 2
  • 18
  • 23