How do I remove blank elements from an AstroPy table?

Question

I'm trying to remove any elements from an astropy table that have blank fields. However, all the help I have found so far just tells me how to replace it. I have tried replacing these blank fields (denoted by '--') with zeroes. However, whenever I try to filter them out with traditional python loops, they stay put. Is there no way I can remove rows that have blank fields in them?

from astropy.io import ascii

dat = ascii.read('exorgdatabase.txt')
dat['MSINI'].fill_value = 0
print(dat['MSINI'])
dat['PER'].fill_value = 0
print(dat['PER'])
newdat = dat.filled()
print(newdat)


while '0' in newdat['MSINI']:
    newdat.remove('0')

print(newdat['MSINI'])

Here's the output I get:

 MSINI  
--------
mjupiter
      --
      --
      --
0.310432
      --
      --
      --
 7.65457
      --
     ...
      --
      --
      --
      --
      --
      --
      --
      --
      --
      --
      --
Length = 5455 rows
    PER    
-----------
        day
   7.958203
 3.27346074
19.12947337
  10.290994
  27.495606
   9.478522
 5.03728015
   2.243752
  7.8125124
        ...
   7.227407
  91.069934
 366.084069
  414.45008
 5.43099314
  328.32211
  381.97977
  67.412998
 2.08802799
359.8249913
  293.70696
Length = 5455 rows
     NAME      MSINI   ...                      FIRSTURL                    
------------- -------- ... -------------------------------------------------
          N/A mjupiter ...                                               N/A
 Kepler-107 d        0 ... http://adsabs.harvard.edu/abs/2014arXiv1402.6534R
Kepler-1049 b        0 ... http://adsabs.harvard.edu/abs/2016ApJ...822...86M
 Kepler-813 b        0 ... http://adsabs.harvard.edu/abs/2016ApJ...822...86M
 Kepler-427 b 0.310432 ... http://adsabs.harvard.edu/abs/2010Sci...327..977B
Kepler-1056 b        0 ... http://adsabs.harvard.edu/abs/2016ApJ...822...86M
Kepler-1165 b        0 ... http://adsabs.harvard.edu/abs/2016ApJ...822...86M
Kepler-1104 b        0 ... http://adsabs.harvard.edu/abs/2016ApJ...822...86M
    WASP-14 b  7.65457 ... http://adsabs.harvard.edu/abs/2009MNRAS.392.1532J
  Kepler-50 b        0 ... http://adsabs.harvard.edu/abs/2011ApJ...736...19B
          ...      ... ...                                               ...
  KOI 2369.03        0 ...                                               N/A
  KOI 7572.01        0 ...                                               N/A
  KOI 7587.01        0 ...                                               N/A
  KOI 7589.01        0 ...                                               N/A
  KOI 2859.05        0 ...                                               N/A
  KOI 7591.01        0 ...                                               N/A
  KOI 7592.01        0 ...                                               N/A
  KOI 7593.01        0 ...                                               N/A
  KOI 7594.01        0 ...                                               N/A
  KOI 7596.01        0 ...                                               N/A
  KOI 7599.01        0 ...                                                 e
Length = 5455 rows
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-118-c200fae23235> in <module>()
     23 
     24 while '0' in newdat['MSINI']:
---> 25     newdat.remove('0')
     26 
     27 print(newdat['MSINI'])

AttributeError: 'Table' object has no attribute 'remove'

what exactly are you doing? Could you add the imports, a shortened version of your data (copy&pastable) and the outputs of your `print`? Otherwise it will be impossible to help you without guessing. — MSeifert, Aug 24 '16 at 12:09

score 2 · Answer 1 · answered Aug 24 '16 at 14:04

2

You cannot!

Removing single fields from a astropy.table.Table is impossible because it is based on numpy. Therefore you can only remove whole columns or rows but not single elements.

For example you have such a Table:

>>> from astropy.table import Table, MaskedColumn, Column

>>> a = MaskedColumn([1, 2], name='a', mask=[False, True], dtype='i4')
>>> b = Column([3, 4], name='b', dtype='i8')
>>> tbl = Table([a, b])
>>> tbl
 a   b 
--- ---
  1   3
 --   4

you can replace masked values with

>>> tbl = tbl.filled(0)
>>> tbl
 a   b 
--- ---
  1   3
  0   4

and for example to remove all rows where a is 0:

>>> tbl[tbl['a'] != 0]
 a   b 
--- ---
  1   3

or without filling it by accessing the mask:

tbl[~tbl['a'].mask]
 a   b 
--- ---
  1   3

answered Aug 24 '16 at 14:04

MSeifert

145,886
38
333
352

I managed to remove NaN elements after converting to pandas. However, that brings me to a new question. How do I get pandas to only remove elements with NaNs in certain columns? – Dewy Bukia-Peters Aug 24 '16 at 14:29
I have no idea about how to do it in `pandas`, sorry. But in order to get more feedback you can open a **new** question specifically asking about how to do it in pandas (if there is no such question already). – MSeifert Aug 24 '16 at 14:33
@Michele He mentioned "certain" columns but that `dropna` would drop all rows that contained *any* nan. – MSeifert Jan 25 '18 at 20:12

How do I remove blank elements from an AstroPy table?

1 Answers1