2

Firstly I am new to this forum so please excuse as to any mistake I am doing in posting here. I would be glad if you could point me my mistakes out and I will make sure I don't repeat them when I post any thing else.

Task: Converting ASCII Data Files (UCAC 4 Star Catalog) to netCDF Format using Python. Only output some fixed number of columns from ASCII file to netCDF file.

Problem: Traceback (most recent call last): File "D:\Work 1\FINAL\New Try\txt2nc.py", line 51, in vic_runoff[ra,spd,:,:] = output; File "netCDF4.pyx", line 2821, in netCDF4.Variable.setitem (netCDF4.c:35204) File "C:\Python27\lib\site-packages\netCDF4_utils.py", line 187, in _StartCountStride ee = range(start,stop,step) File "C:\Python27\lib\site-packages\numpy\ma\core.py", line 4102, in int raise MaskError('Cannot convert masked element to a Python int.') MaskError: Cannot convert masked element to a Python int.**

Thanks in advance. Any help is appreciated!

    from __future__ import division
    from netCDF4 import Dataset
    import numpy as np
    import os

    PATH = 'D:\\Internship Work 1\\Alok Data\\ASCII'
    LL = np.loadtxt('%s\\4uc001.txt' %PATH, delimiter='|', usecols = 
     (0,1,2,3), skiprows=0);
    LL = LL[:,:]

    # NC file setup
    root_grp = Dataset('%s\\4uc001.nc' %PATH, 'w', format='NETCDF4')
    root_grp.description = 'Star Catalog UCAC Data'

    # dimensions
    ra = root_grp.createDimension('ra', 32)
    spd = root_grp.createDimension('spd', 80)
    magm = root_grp.createDimension('magm', 96)
    maga = root_grp.createDimension('maga', 120)


    # variables
    ra = root_grp.createVariable('ra', np.byte, ('ra',))
    spd = root_grp.createVariable('spd', np.byte, ('spd',))
    magm = root_grp.createVariable('magm', np.byte, ('magm'),)
    maga = root_grp.createVariable('maga', np.byte, ('maga'),)
    vic_runoff = root_grp.createVariable('vic_runoff', np.byte, ('ra',
     'spd', 'magm', 'maga',))

    ra.units = 'mas'
    spd.units = 'mas'
    magm.units = 'millimag'
    maga.units = 'millimag'

    for ra in enumerate(ra):
      tempstore = np.zeros((206,4),int)


    output_filename = 'D:\\Internship Work 1\\Alok Data\\ASCII\\4uc001.txt'

    output = np.loadtxt(output_filename,delimiter='|',usecols = (0,1,2,3))

    tempstore[:,:] = output # ensembles x months
    vic_runoff[ra,spd,:,:] = tempstore[:,:] # write all ensembles to netcdf

    print('work done')
Alok Dadlani
  • 21
  • 1

1 Answers1

2

A few comments that will hopefully help you:

In vic_runoff[ra,spd,:,:], spd is a netCDF4 variable. It can not be used as an indexer. Try vic_runoff[ra,0,:,:] = tempstore[:,:] to see if that fixes your problem.

Also, you should consider using pandas.read_csv or read_table to read your ASCII files. There's a SO post around here documenting the performance improvement over numpy.loadtxt

Lastly, if you went the pandas route, you may consider using xarray for the writing to netCDF. xarray easily allows you to convert from pandas to a xarray.Dataset then you're whole netCDF file is written with a quick Dataset.to_netcdf() call.

jhamman
  • 5,867
  • 19
  • 39