python: eliminate positions of nan in multiple lists

Question

I am coming from IDL and trying to find and eliminate the NaNs in two lists of data. lets say there is a NaN in position 5 for list A but not list B. I need position 5 to be removed in both lists. Like so...

A = [1, NaN, 3, 4, NaN, 6, 7, 8, 9]

B = [1', 2', 3', NaN, 5', 6', 7', 8', NaN]

A_new = [1 , 3 , 6 , 7 , 8 ]

B_new = [1', 3', 6', 7', 8']

Here is the IDL code that works fine. I just need it translated to python and I am stumped.

;Removes the NANs

loc = where((Finite(pdcsapflux) EQ 1)and(Finite(time) EQ 1))

flux_nonan = pdcsapflux(loc)

time_nonan = time(loc)

Thanks!!

one prime. Just a place holder for some sort of data. I wanted to differentiate it from 1 in A. It can be any sort of finite value — Thomas Cannon, Feb 16 '15 at 04:04
If you're coming from IDL, you probably want to be using [`numpy`](http://numpy.org) and [`pandas`](http://pandas.pydata.org) instead of pure Python lists. — DSM, Feb 16 '15 at 04:07
@ThomasCannon Did you mean `None` or something like `1/0` when you say `NaN`? — satoru, Feb 16 '15 at 04:13

score 1 · Answer 1 · answered Feb 16 '15 at 04:17

1

A = [1, None, 3, 4, None, 6, 7, 8, 9]
B = [1, 2, 3, None, 5, 6, 7, 8, None]
print zip(*[
    (a, b) for a, b in zip(A, B)
    if a is not None and b is not None
])

Documentation of zip

answered Feb 16 '15 at 04:17

satoru

31,822
31
91
141

score 1 · Answer 2 · answered Feb 18 '15 at 17:31

Following on @DSM's suggestion, if you're coming from IDL, you probably want to use numpy with actual arrays, not lists. The direct replacement for the IDL code using numpy is more like the following:

import numpy.random as ran
import numpy as np
arr = ran.rand(10)  # create some fake data
arr[[1,3,5]] = np.nan  # add some "bad" values
arr2 = arr[np.where(~np.isnan(arr))]

Hope this helps.

score 0 · Accepted Answer · answered Feb 16 '15 at 04:08

I dont know if python has NaN, just assume that is None.

ta, tb = [], []
for i in range(min(len(a), len(b))):
    if a[i] is None or b[i] is None:
        continue
    ta.append(a[i])
    tb.append(b[i])

ta, tb is your output of a,b at last you should append the rest item of the longer list.

score 0 · Answer 4 · answered Mar 18 '15 at 11:52

If you are using numpy you can also do it without where function (usually very slow if you work with big matrices)

import numpy as np 
A = np.array([1, NaN, 3, 4, NaN, 6, 7, 8, 9])
B = np.array([1, 2, 3, NaN, 5, 6, 7, 8, NaN])
A_new = A[~np.isnan(A)]
B_new = B[~np.isnan(B)]

python: eliminate positions of nan in multiple lists

4 Answers4