Slicing numpy recarray at "empty" rows

Question

I created a numpy.recarray from a .csv-Inputfile using the csv2rec()-Method. The Inputfile and consequently the recarray have empty rows with no data (resp. nan-values). I want to slice this recarray at the nan-rows into multiple sub-arrays, excluding the nan-rows in the final arrays as shown below.

Original recarray with 2 columns:

[(1,2)
(2,2)
(nan,nan)
(nan,nan)
(4,4)
(4,3)]

2 sub-arrays without nan-values:

[(1,2)
(2,2)]

and

[(4,4)
(4,3)]

I know this could be managed using a loop but maybe there's a simpler and more elegant way? Additionally: Is it possible to keep the header-information of each column so I can refer to the columns by the parameter-name and not only the col-index after the slicing?

Check out Pandas, it's well suited to this kind of thing: http://pandas.pydata.org/pandas-docs/dev/missing_data.html — YXD, Jul 01 '13 at 09:31

Saullo G. P. Castro · Accepted Answer · 2013-07-01T10:35:13.657

For a 2D-array:

a[~np.all(np.isnan(a),axis=1)]

For a structured array (recarray) you can do this:

def remove_nan(a, split=True):
    cols = [i[0] for i in eval(str(a.dtype))]
    col = cols[0]
    test = ~np.isnan(a[col])
    if not split:
        new_len = len(a[col][test])
        new = np.empty((new_len,), dtype=a.dtype)
        for col in cols:
            new[col] = a[col][~np.isnan(a[col])]
        return new
    else:
        indices = [i for i in xrange(len(a)-1) if test[i+1]!=test[i]]
        return [i for i in np.split(a, indices) if not np.isnan(i[col][0])]

To get only the lines without nan use split=False. Example:

a = np.array([(1,2),(2,2),(nan,nan),(nan,nan),(4,4),(4,3)], dtype=[('test',float),('col2',float)])

remove_nan(a)

#[array([(1.0, 2.0), (2.0, 2.0)],
#      dtype=[('test', '<f8'), ('col2', '<f8')]),
# array([(4.0, 4.0), (4.0, 3.0)],
#      dtype=[('test', '<f8'), ('col2', '<f8')])]

Maybe I did not state my intention clearly: The main aim of the slicing was to get the subarrays as "individuals", not to get rid of the nan-rows. — ala, Jul 01 '13 at 09:39
@ sgpc Thank you, this method works fine and it's much shorter than my version. — ala, Jul 01 '13 at 12:42

score 0 · Answer 2 · answered Jul 01 '13 at 09:28

0

If you just wish to get rid of the blanks, rather than slice on them, then just compress your array with the selection criteria being a check for not nan. Hint, nan <> nan.

If you really wish to slice at the nans then use a loop of some such to generate a list of the Non-Nan indexes and then use choose to generate the sub-arrays - they should retain the col names that way.

answered Jul 01 '13 at 09:28

Steve Barnes

27,618
6
63
73

I do not necessarily need to slice the array but at least I'd have to know the indices of start- and end of each non-nan-section, so I could refer to the sections by start- and end-index (this would even be possible without the slicing). – ala Jul 01 '13 at 09:34

score 0 · Answer 3 · answered Jul 02 '13 at 00:27

You can use scipy.ndimage.label to get regions in an array of 0's and 1's:

>>> import numpy as np
>>> from scipy import ndimage
>>> nan = np.nan
>>> a = np.array([(1,2),(2,2),(nan,nan),(nan,nan),(4,4),(4,3)], dtype=[('test',float),('col2',float)])
>>> non_nan = np.logical_not(np.isnan(a['test'])).astype(int)
>>> labeled_array, num_features = ndimage.label(non_nan)
>>> for region in range(1,num_features+1):
...     #m = a[np.where(labeled_array==region)]
...     m = a[labeled_array==region]
...     print(region)
...     print(m)
...     print(m['col2'])
...
1
[(1.0, 2.0) (2.0, 2.0)]
[ 2.  2.]
2
[(4.0, 4.0) (4.0, 3.0)]
[ 4.  3.]

If you know you will always have two regions then you don't need the loop and just refer to:

m1 = a[labeled_array==1]
m2 = a[labeled_array==2]

Slicing numpy recarray at "empty" rows

3 Answers3