Write boolean structured arrays with PyFITS

Question

I would like to write a Boolean structured array with PyFITS in a FITS file. I had some issues. Here is a simple example.

I create the test dictionary and transform it into a structured array.

In [241]: test = {'p':np.array([True]*10+[False]*10,dtype='b')}
In [242]: test = np.core.records.fromarrays(list(test.values()), names=list(test.keys()))

Here it is the test structured array I would like to print in a .fit file.

In [243]: test
Out[243]: 
rec.array([(1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), (1,), (0,),
   (0,), (0,), (0,), (0,), (0,), (0,), (0,), (0,), (0,)], 
  dtype=[('p', 'i1')])

I print test in a fit file using pyfits

In [244]: pyfits.writeto('./test.fit',test,clobber=True)

In [245]: d = pyfits.open('./test.fit')

In [246]: d = d[1].data

However, all entries are now set to the False value, as follows:

In [247]: d  
Out[247]: 
FITS_rec([(False), (False), (False), (False), (False), (False), (False),
   (False), (False), (False), (False), (False), (False), (False),
   (False), (False), (False), (False), (False), (False)], 
  dtype=[('p', 'i1')])

Furthermore, it seems that the original test array is also somehow modified by pyfits.

In [248]: prova
Out[248]: 
rec.array([(70,), (70,), (70,), (70,), (70,), (70,), (70,), (70,), (70,),
       (70,), (70,), (70,), (70,), (70,), (70,), (70,), (70,), (70,),
       (70,), (70,)], 
      dtype=[('p', 'i1')])

Could you please help me to solve this issue? Thank you very much!

Iguananaut · Accepted Answer · 2015-09-03T20:06:01.743

Boolean columns in FITS are poorly understood, in large part due to their unusual representation (which uses the ASCII characters 'T' and 'F' to store true and false values, hence the 70s you're getting, which are ASCII 'F's.

Nevertheless, I've put a lot of work in the past into making this work correctly, so that even if you pass in an array of 0's and 1's it should infer what you meant by that. There seems to be a bug here, that the writeto "convenience" function isn't handling the boolean-ish array properly. I was able to make it work like this instead:

>>> hdu = fits.BinTableHDU.from_columns(test)
>>> hdu.writeto('test.fits', clobber=True)
>>> fits.getdata('test.fits')
FITS_rec([(True), (True), (True), (True), (True), (True), (True), (True),
       (True), (True), (False), (False), (False), (False), (False),
       (False), (False), (False), (False), (False)], 
      dtype=[('p', 'i1')])

What you did in the first place probably should have worked. Though in general I would recommend using dtype='?' or equivalently dtype=bool explicitly if you want the type to be correctly guessed as boolean (otherwise there's some ambiguity as to whether or not you actually wanted bytes).

Update: There's also an old bug report about the same problem here: https://github.com/astropy/astropy/issues/1901 Apparently I tried to solve this a while ago but got fed up with the ambiguities. Which is strange because I thought I did fix this at one point... In any case if you explicitly make your arrays bool dtype and use the .from_columns method as demonstrated above it should work. I'll see about revisiting some of these bugs.

Write boolean structured arrays with PyFITS

1 Answers1