4

I have downloaded TRMM monthly precipitation rate in netCDF format from 1998 -2016, so approximately more than 200 files.The names of these files are 3B43.19980101.7.HDF.nc 3B43.19980201.7.HDF.nc 3B43.19980301.7.HDF.nc , and so on. I would like to concatenate all of these files into a single netCDF. I've tried using the NCO operator "ncrcat" which should be able to concatenate a very long series of files along the record dimension, in this case time, but so far no luck. I tried at first simple with only 2 files

ncrcat -O -h 3B43.19980101.7.HDF.nc 3B43.19980201.7.HDF.nc out.nc

got

ERROR: no variable fit criteria for processing

so then I tried 

ncks --mk_rec_dmn time 3B43.19980101.7.HDF.nc TD.3B43.19980101.7.HDF.nc
ncks --mk_rec_dmn time 3B43.19980201.7.HDF.nc TD.3B43.19980201.7.HDF.nc

I tried again with 

ncrcat -O -h TD.3B43.19980101.7.HDF.nc TD.3B43.19980201.7.HDF.nc out.nc

still got same error

ERROR: no variable fit criteria for processing

Is there an easier way to doing this with 200+ files? A script that I can follow? I am new to all this so please be gentle.

Any help would be greatly appreciated. I am using Windows 7 x86.

Cyrus Mohammadian
  • 4,982
  • 6
  • 33
  • 62
marie_r
  • 67
  • 1
  • 7
  • I have access to the PPS TRMM FTP server. Can you send the link to the 3B43.19980101.7.HDF.nc files you are accessing? I can take a look. – Eric Bridger Oct 05 '16 at 22:04
  • Is there a way to share these two files with us? That will help to diagnose the issue. Could you do an `ncdump -h [filename]` for one of the files? – N1B4 Oct 05 '16 at 23:07
  • @EricBridger I downloaded the files from http://mirador.gsfc.nasa.gov/ using the coordinates (15.04,-90.05),(15.31,-89.24) – marie_r Oct 12 '16 at 17:10
  • @N1B4 Here is a link: https://www.dropbox.com/sh/93xv98q3g3cur8v/AAA0TCCcw2mdDWo7hp6HvhaYa?dl=0 – marie_r Oct 12 '16 at 17:26
  • @N1B4 For some reason ncdump isn't working for me. After installing, if I go to the NCO folder I can find: ncap2, ncatted, ncbo, ncea, ncecat, ncflint, ncks, ncpdq, ncra, ncrcat, ncrename, & ncwa. But no ncdump... – marie_r Oct 12 '16 at 17:31
  • @marie_r I did find the files and spent some time because this is a dataset I'd like to use. My verdict was that it's not possible with nco tools alone. Basic issue is the files do not have a time variable in them. You need to create a time variable first, setting the value based on the file name. I would use the python NetCDF4 library, don't know R. Then you could create a time dimension, mk_rec_dmn time, then use ncrcat. – Eric Bridger Oct 12 '16 at 18:49
  • Any idea on how to do it using Python? – marie_r Oct 17 '16 at 14:29
  • It is completely possible to do this with NCO. You have the right strategy, and simply need to add a time dimension (a time variable is not necessary, though it wouldn't hurt either). Add a time dimension with – Charlie Zender Oct 23 '16 at 02:02

2 Answers2

3

It is completely possible to do this with NCO. I looked at your input files and they simply lack a time dimension, so ncrcat fails. Add a time dimension with

ncecat -u time in.nc out.nc

Then use ncrcat as you say above. p.s. I have changed the ncrcat and ncra error messages to be more explicit about how to do this. Previously the HINTs only applied to cases where the file already had the dimension, but it was fixed. Your files did not have a time dimension, so the ncks command you issued had no effect.

Edit to show loops:

To do this or anything like it in a loop use a construct like

for fl in `ls trmm*.nc`; do
    ncecat -u time ${fl} ${fl/trmm/trmm_new} # Base output name in input name
    ... # more processing
done

The NCO manual has many examples of using file loops.

Charlie Zender
  • 5,929
  • 14
  • 19
  • thanks! I think this might have worked. Now my final question is if there's a way to do this in a loop. I have more than 200 files that I would like to add a time dimension to then concatenate together. – marie_r Oct 24 '16 at 16:51
2

In R, you can do this by reading in all the data, combining into one large 3d array (latxlonxtime). For example, array[,,1] would be the latxlon grid for Jan 1998. This can then be saved as a .rds format for further use in R, or saved as a netCDF file, which I won't cover but there are tutorials for saving R arrays as .nc files online.

First, make a .csv file that contains a single column of all the filenames you downloaded. One easy way is to ctrl-C the output from typing 'ls' in terminal into an excel sheet. The code below reads in those files one by one, adding each to the array.

library(ncdf4)
library(abind)
filenames=read.csv('TRMM.filenames.csv',head=F) #read in filenames
filenames=as.character(filenames[,1]) #convert to 'character' format

n.lon=192 #input the correct #'s here, must be the same for all files
n.lat=94

NA.matrix=matrix(rep(NA,n.lon*n.lat),nrow=n.lon) #used to initialize
prcp=array(NA.matrix,c(n.lon,n.lat,1)) #n.lonxn.latx1 array of NA's to initialize
for (i in 1:length(filenames)){
  ncdata=nc_open(filenames[i]) #read in file i, assuming files are in same location as filenames.csv/your current working directory
  #ncdata=nc_open(paste(data.dir,filenames[i],sep="")) #if your data is in another directory than the filenames.csv file, you could read it in with this line instead
  nc=ncvar_get(ncdata,"precip") #check the .nc files to see what the variable name actually is; this reads in the variable "precip"
  prcp=abind(prcp,nc)
}
prcp=prcp[,,-1] #remove the NA.matrix used to initialize

dim(prcp) #check that the lonxlatxtime dimensions make sense
saveRDS(prcp,'TRMM.all.rds') #save as .rds file, or proceed to save it as .nc file, which takes a bit more work
Justina Pinch
  • 367
  • 1
  • 7
  • What do you mean by n.lon = 192 & n.lat = 94? How do I check my correct #? – marie_r Oct 12 '16 at 17:12
  • The longitude and latitude dimensions of the datasets. In terminal if you have netCDF installed and loaded you can do `ncdump -h filename` and it will print summary info. Otherwise in R open up one of your files: `filename="3B43.19980101.7.HDF.nc"; ncdata=nc_open(filename); print(ncdata)` This gives you a summary of the file, including the 'size' for the longitude and latitude dimensions. – Justina Pinch Oct 12 '16 at 20:46
  • Thanks a lot Jack! Using your script I was able to create an rds file. However, I'm really interested in saving it as an nc file. Any idea where could I find more info on this matter? – marie_r Oct 17 '16 at 14:31