Creating/concatenate multi-dimensional NetCDF in R

Question

Using this as a source : How to concatenate monthly TRMM netCDF files into a single netCDF file using NCO or R on windows 7?

install.packages("ncdf4")
library(ncdf4)
install.packages("abind")
library(abind)
install.packages("RNetCDF")
library(RNetCDF)
install.packages("ncdf.tools")
library(ncdf.tools)
filenames=read.csv('TRMM.filenames.csv',head=F) 
filenames=as.character(filenames[,1]) 
n.lon=4 
n.lat=7 
NA.matrix=matrix(rep(NA,n.lon*n.lat),nrow=n.lon) 
prcp=array(NA.matrix,c(n.lon,n.lat,1)) 


for (i in 1:length(filenames)){ncdata=nc_open(filenames[i])
+ nc=ncvar_get(ncdata,"precipitation") prcp=abind(prcp,nc)}

prcp=prcp[,,-1] 
dim(prcp) 
saveRDS(prcp,'TRMM.all.rds')

I was able to create an rds file. However, I'm really interested in saving it as an nc file. I tried creating a new netCDF file with a 12 steps (one for each month) time dimension by :

dimx <- ncdim_def( "Lon", "degreesE", as.double(-90:-87))
dimy <- ncdim_def( "Lat", "degreesN", as.double(14:16))
dimTime <- ncdim_def( "Time", "months", 1:12, unlim=TRUE )
dimlist<-list(dimx,dimy,dimTime)
precip.ncvar<- ncvar_def("Precip", "mm/hr", dimlist, -1, longname="Precipitation", prec="float")
precip.nccreate<- nc_create( "precip.nccreate.nc", precip.ncvar, force_v4=FALSE, verbose=FALSE )
nc_close(precip.nccreate)

Now the challenge is to add the precipitation data to each month.
Following the first script I tried using the ncvar_put function without success.

filenames1=read.csv('TRMM.filenames.csv',head=F) 
filenames1=as.character(filenames1[,1]) 

for (i in 1:length(filenames1)){ncdata1=nc_open(filenames1[i])
nc1=ncvar_get(ncdata1,"precipitation") 
prcp1=abind(prcp1,nc1)}

n.lon1=4 
n.lat1=7

data2d<-(4*7)

for (i in 1:length(filenames1))
  ncvar_put( precip.nccreate, precip.ncvar, data2d, start=c(1), count=c(1) )

precip.nccreate<- nc_create( "precip.nccreate.nc", precip.ncvar, force_v4=FALSE, verbose=FALSE )

To which I got

Error in ncvar_put(precip.nccreate, precip.ncvar, data2d, start = c(1), : object 'precip.nccreate' not found

Error in nc_create("precip.nccreate.nc", precip.ncvar, force_v4 = FALSE, : object 'precip.ncvar' not found

Anyway, I guess I'm just trying to find an easy way to concatenate multiple netcdf files into a single netcdf.

Thanks

Is there a way I could concatenate multiple netcdf into a single netcdf using python? — marie_r, Oct 19 '16 at 19:38
@marie_r As I mentioned in my comments on your previous question, there is a way to do it using python. Note using the @ syntax would have notified me of you last comment. Can you run python? — Eric Bridger, Oct 19 '16 at 20:49
@ericbridger Sure I can. Could you tell me how to do it using python? — marie_r, Oct 24 '16 at 12:32

score 1 · Accepted Answer · answered Oct 23 '16 at 01:55

1

Since the time dimension is unlimited, you can use NCO's ncrcat command, e.g.,

ncrcat in1.nc in2.nc ... out.nc
ncrcat in*.nc out.nc

answered Oct 23 '16 at 01:55

Charlie Zender

5,929
14
19

score 0 · Answer 2 · answered Oct 25 '16 at 15:40

The TRMM files have no time variable or dimension. I could not get NCO tools to handle that situation, but I'm no expert. My solution was to use python to create a copy of the file while adding a time dimension and a time value based on the input file name. Then using ncrcat as @CharlieZender suggests. Sorry for the long script.

#! /usr/bin/env python
import os
import glob
import netCDF4
import datetime

# get all the TRMM files with file names like, add a time dimension and time variable value from the file name.
# E.G. 3B43.19980101.7.HDF.nc, 3B43.19980201.7.HDF.nc, 3B43.20160701.7.HDF.nc
# Creates a list of monthly files which can be concatenated by NCO Tools ncrcat command.

# assumes all the 3B43.*.nc files are in the subdir files/
# creates out_3B43.*.nc in the current directory
infiles = glob.iglob( "files/*.nc" )

# add time dimension to these gridded variables. we only care about precipiation.
#wanted_vars = ['precipitation', 'relativeError', 'gaugeRelativeWeighting']
wanted_vars = ['precipitation']

for infile in sorted(infiles):
  try:
    nci = netCDF4.Dataset(infile, 'r')
  except RuntimeError:
    print "ERROR: Could not open " + infile
    exit()

  ofile = 'out_' + os.path.basename(infile)

  try:
    nco = netCDF4.Dataset(ofile,'w',clobber=True, format=nci.data_model)
  except RuntimeError:
    print "ERROR: Could not open " + ofile
    exit()

  # copy global attributes
  in_g_attdict = nci.__dict__
  # copy dimensions
  nco.setncatts(in_g_attdict) 
  for dname, d in nci.dimensions.iteritems():
    if d.isunlimited():
      d_size = None
    else:
      d_size = len(d)
    nco.createDimension(dname, d_size)
  # Add a time dimension, as unlimited, None means unlimited.
  nco.createDimension('time', None)

  # Get YYYYMMDD from file name you could use dy = 15 to reflect that these are monthly files.
  file_date = infile.split('.')[1]
  yr = file_date[0:4]
  mo = file_date[4:6]
  dy = file_date[6:]
  odt = datetime.datetime(year=int(yr), month=int(mo), day=int(dy))

  # create the time variable. Note: the time variable must go first to be the outer or record dimension for ncrcat
  time_v = nco.createVariable('time', 'int32', ('time'))
  time_v.setncattr('standard_name', 'time') 
  time_v.setncattr('long_name', 'reference time of data field') 
  time_v.setncattr('axis', 'T') 
  # for time_urnits, I've used seconds since the epoch. But I'm guessing that 'days since XXX' should work as well. But harded to
  time_v.setncattr('units', 'seconds since 1970-01-01 00:00:00 UTC')
  # calculate the number.
  ntime = netCDF4.date2num(odt, time_v.units)
  time_v[:] = ntime

  # Copy variables, skip the wanted_vars, for now
  for iname, ivar in nci.variables.iteritems():
    if iname in wanted_vars:
      continue
    ovar = nco.createVariable(iname, ivar.dtype, ivar.dimensions)
    iattdict = ivar.__dict__
    ovar.setncatts(iattdict) 
    ovar[:] = ivar[:]
  # now copy the wanted 2-D gridded variables, adding a time dimension, as axis 0
  for data_var_name in wanted_vars:
    ivar = nci.variables[data_var_name]
    # original precipation variable is dimensioned by nlon,nlat
    ovar = nco.createVariable(data_var_name, ivar.dtype, ('time', 'nlon', 'nlat') )
    iattdict = ivar.__dict__
    ovar.setncatts(iattdict) 
    ovar[0,:,:] = ivar[:]

  nci.close()
  nco.close()

exit()

This will leave you with out_xxxxx.nc files in the current directory and ncrcat can be used.

ncrcat -H -h out_3B43.19980101.7.HDF.nc out_3B43.19980201.7.HDF.nc final.nc

score 0 · Answer 3 · answered Apr 08 '17 at 22:51

0

Merging in time is done in this way with CDO:

cdo mergetime in1.nc in2.nc ... in12.nc out.nc

I use cdo frequently to manipulate TRMM files.

answered Apr 08 '17 at 22:51

ClimateUnboxed

7,106
3
41
86

Creating/concatenate multi-dimensional NetCDF in R

3 Answers3