.FITS file/table to csv?

Question

Any suggestions on how to convert a .fits file (or table) to a csv? I tried to and the code was running for over 12 hours. I currently used pandas and read the table and tried to save as csv:

  dr_file = get_pkg_data_filename('/users/...data.fits')
  events = Table.read(dr_file, hdu=1)
  df = pd.DataFrame(events)
  df.to_csv('data.csv')

Is it an astronomy file? Are you planning on sharing it via Dropbox or Google Drive or similar so folks can assist you? — Mark Setchell, Mar 01 '21 at 13:42
@MarkSetchell Yes it is. I planned to save to my computer and cross match with other data I have. — Lily, Mar 01 '21 at 13:51
You *could* use **vips** in the Terminal as simply as this `vips im_vips2csv INPUT.fits OUTPUT.csv` Takes around 70 milliseconds ;-) — Mark Setchell, Mar 01 '21 at 14:12
@MarkSetchell taking longer than 1 second for me... the fits file is 4gb. Is this file size too large to handle? — Lily, Mar 01 '21 at 14:19
CSV s ridiculously inefficient for images, so it may take a while if your image has large height/width and is floating point. What are the dimensions? — Mark Setchell, Mar 01 '21 at 14:22
You can use the `vips` Python bindings and load it directly if you want to avoid CSV. Here https://pypi.org/project/pyvips/ — Mark Setchell, Mar 01 '21 at 14:23
@MarkSetchell dimensions: 733901R x 224C . This is a table of data, dr17 for reference. — Lily, Mar 01 '21 at 14:27
Or `wand` should be able to load it directly too. https://docs.wand-py.org/en/0.6.6/ — Mark Setchell, Mar 01 '21 at 14:32
That's actually 164 million pixels which might each be say 16-40 bytes when saved as ASCII in a CSV making 7+ GB... — Mark Setchell, Mar 01 '21 at 14:35

score 2 · Answer 1 · answered Aug 25 '21 at 03:54

2

@Iguananaut is correct; here is a code sample as a template:

from astropy.io import fits
from astropy.table import Table

# set memmap=True for large files
with fits.open("my_file.fits", memmap=True):

    # select the HDU you want
    hdu = hdu_list[1]
    
    # read into an astropy Table object
    table = Table(hdu.data)

    # write to a CSV file
    data.write('path/to/output/my_file.ecsv', delimiter='\t', format='ascii')

You certainly don't want to do this for images, but for tables an intermediate CSV format can be useful in some cases, e.g. importing into a database.

answered Aug 25 '21 at 03:54

Demitri

13,134
4
40
41

Hey, this template works but it merges all the columns on the csv file together. Is there a way to sperate the columns – Jacob Dec 20 '21 at 15:54
Merges in what way? You can set the delimiter to any character you want. If you can point to an example file and your code on a GitHub gist I'd be happy to take a look. – Demitri Dec 29 '21 at 20:54

score 2 · Answer 2 · answered Nov 29 '21 at 13:38

2

Bit late to the party, but TOPCAT also has the option to save fits tables to other formats including csv. Fast and easy.

answered Nov 29 '21 at 13:38

user17541936

21
1

score 1 · Answer 3 · answered Mar 02 '21 at 10:42

Astropy tables can already be written directly to CSV. No need to put it in a Pandas DataFrame first. Astropy has extensive CSV and ASCII table functionalities geared toward formats used in astronomy. See https://docs.astropy.org/en/stable/io/ascii/write.html#table-or-numpy-structured-array

As others have pointed out, however, dumping a very large table to a text format may be inappropriate since it could be enormous in size and difficult for most software to process efficiently. You should consider other binary formats (several of which, such as HD5, are also supported by Astropy) or you should extract only the specific data you need to share, or just share the FITS file.

.FITS file/table to csv?

3 Answers3