Concatenate two nc4 files of different time span

Question

Assuming two netCDF files (nc4 in my case), each of one variable with 3 dimensions: latitude, longitude and time (year). They are of different time spans (e.g. 1700-2005 and 2005-2100 - they overlap at 2005). How can I concatenate them to obtain one nc4 file spanning 1700-2100?

Simply trying the NCO operator ncrcat file1.cn4 file2.nc4 result.nc4 gives a file that spans [1700-2005, 1700-1795]. When running the line it warns: nco_cln_clc_dff<><> failed to initialize UDUnits2 library. According to this post, I should not worry.

I also used appending as described in the NCO manual ncrcat --rec_apn file2.nc4 file1.cn4. I get a resulting time span [2005-2310,2005-2100] and a warning: ncrcat: WARNING Intra-file non-monotonicity. Record coordinate “time” does not monotonically decrease between input file file2.nc4 record indices: 94,95 output file1.nc4 record indices 400,401 ... (the indices are less important here as I get the warning for every one of them)

Note that I could concatenate without the time span issue with CDO on Linux: cdo mergetime file1.cn4 file2.nc4 result.nc4 (or cdo -z zip_3 mergetime file1.cn4 file2.nc4 result.nc4 to get a level 3 of compression). Note also that I had to use export SKIP_SAME_TIME=1 before calling cdo mergetimein order to deal with the overlapping 2005 year (will take only the first occurrence).

With R I tried the following:

library(ncdf4)
library(ncdf.tools)
ncFile1 <- nc_open("C://file1.nc4")
nc1 <-ncvar_get(ncFile1)
ncFile2 <- nc_open("C://file2.nc4")
nc2 <-ncvar_get(ncFile2)
transNcdfMerge(c(nc1, nc2), target.name = "my_test.nc4")

I let it run for almost 2 hours and then I stopped the run. Since I have to do this for hundreds of nc4 files, I can't wait that long. Didn't yell any error while running but I am not sure if the code is correct.

I use Windows 7, 64 bit, 8Gb RAM. And for R - "R version 3.3.0 (2016-05-03)"

Update: printing some metadata of the two files with R

1) file1.nc4

ncdf4::print(file1.nc4)

 1 variables (excluding dimension variables):
    float prop_crop[lon,lat,time]   (Chunking: [720,360,1])  (Compression: shuffle,level 3)
        units: percent
        _FillValue: -9999
        long_name: Proportion of landcover in crops
        missing_value: -9999

 3 dimensions:
    lat  Size:360
        units: degrees_north
        long_name: Latitude
        standard_name: latitude
    lon  Size:720
        units: degrees_east
        long_name: Longitude
        standard_name: longitude
    time  Size:306   *** is unlimited ***
        long_name: Time
        standard_name: time
        calendar: proleptic_gregorian
        units: years since 1700-01-01 00:00:00

2) file2.nc4

ncdf4::print(file2.nc4)

 1 variables (excluding dimension variables):
    float prop_crop[lon,lat,time]   (Chunking: [720,360,1])  (Compression: shuffle,level 3)
        units: percent
        _FillValue: -9999
        long_name: Proportion of landcover in crops
        missing_value: -9999

 3 dimensions:
    lat  Size:360
        units: degrees_north
        long_name: Latitude
        standard_name: latitude
    lon  Size:720
        units: degrees_east
        long_name: Longitude
        standard_name: longitude
    time  Size:96   *** is unlimited ***
        units: years since 2005-01-01 00:00:00
        long_name: Time
        standard_name: time
        calendar: proleptic_gregorian

Hope it helps

Can you share a snippet of the files you're concatenating or at least an ncdump/ncks view of their metadata? — N1B4, Jul 06 '16 at 00:03

score 3 · Answer 1 · answered Jul 06 '16 at 22:09

3

NCO requires UDUnits to perform the date arithmetic (called "rebasing") in the manual. Suggest you install a fully functional NCO with, e.g.,

conda install -c conda-forge nco

then retry your original command.

answered Jul 06 '16 at 22:09

Charlie Zender

5,929
14
19

I installed nco-4.6.0.windows.mvs.exe for Windows from http://nco.sourceforge.net/#Binaries I see that your command is mentioned for Anaconda - do I need to install Anaconda now? – Valentin_Ștefan Jul 06 '16 at 22:31
1

Yes, Anaconda must be installed for "conda install" to work. Unfortunately I don't think conda supports NCO on windows (but you can try). I understand it's hard to build UDUnits-enabled NCO on windows by hand, so all I can suggest is doing your data reduction on Mac or Linux. – Charlie Zender Jul 07 '16 at 23:30
I could manage with Climate Data Operators (CDO) on Linux. Still curious how can I deal with netCDF on Windows. – Valentin_Ștefan Jul 08 '16 at 07:17

score 2 · Answer 2 · answered May 31 '17 at 12:35

Just to mention that if, for any reason, the values in 2005 were not consistent in the two files, and one wanted to retain the overlapping portion from the second file rather than the first, one could use seldate with CDO in order to cut out the relevant portion:

cdo mergetime -seldate,17000101,20041231 file1.nc4 file2.nc4 merged_file.nc

Pedro Vicente · Answer 3 · 2016-07-16T22:31:00.260

nco-4.6.0.windows.mvs.exe is linked with UDunits. To verify, try this

C:\nco>ncks -r

The output should be (note the UDunits section) As far as the error you get, to debug your error, we need to have either the original files or a sample made with ncks/ncdump

NCO netCDF Operators version "4.6.0" built by USER on HOSTNAME at May 12 2016 17:17:59
    ncks version "4.6.0"
    Linked to netCDF library version 4.3.2, compiled Feb 29 2016 01:53:06
    Configuration Option:   Active? Meaning or Reference:
    Check _FillValue        Yes     http://nco.sf.net/nco.html#mss_val
    Check missing_value     No      http://nco.sf.net/nco.html#mss_val
    DAP clients             Yes     http://nco.sf.net/nco.html#dap
    Debugging: Custom       No      Pedantic, bounds checking (slowest execution)
    Debugging: Symbols      No      Produce symbols for debuggers (e.g., dbx, gdb)
    ESMF Library            No      http://nco.sf.net/nco.html#esmf
    GNU Scientific Library  Yes     http://nco.sf.net/nco.html#gsl
    HDF4 support            Unknown http://nco.sf.net/nco.html#hdf4
    Internationalization    No      http://nco.sf.net/nco.html#i18n (pre-alpha)
    MPI parallelization     No      http://nco.sf.net/nco.html#mpi (beta)
    netCDF3 64-bit files    Yes     http://nco.sf.net/nco.html#lfs
    netCDF4/HDF5 available  Yes     http://nco.sf.net/nco.html#nco4
    netCDF4/HDF5 enabled    Yes     http://nco.sf.net/nco.html#nco4
    OpenMP SMP threading    No      http://nco.sf.net/nco.html#omp
    Optimization: run-time  No      Fastest execution possible (slowest compilation)
    Parallel netCDF3        No      http://nco.sf.net/nco.html#pnetcdf (pre-alpha)
    Regular Expressions     No      http://nco.sf.net/nco.html#rx
    Shared libraries built  No      Small, dynamically linked executables
    Static libraries built  No      Large executables with private namespaces
    UDUnits conversions     Yes     http://nco.sf.net/nco.html#udunits
    UDUnits2 conversions    Yes     http://nco.sf.net/nco.html#udunits

Note that your file must have units information, in attributes. This is a sample for a file that has unit information for variable 'lon' located in group 'g1'

netcdf in_grp {
group: g1 { 
   dimensions:
   lon=4;
   float lon(lon);
   lon:units = "degrees_east";  
   data:
    lon=0,90,180,270;
  }
}

The source code distribution of NCO has several sample files that are installed in the Windows NCO version. Try

C:\nco>ncks in_grp.nc -g g1 -v lon
/g1/lon
lon: type NC_FLOAT, 1 dimension, 1 attribute, compressed? no, chunked? no, packed? no
lon size (RAM) = 4*sizeof(NC_FLOAT) = 4*4 = 16 bytes
lon dimension 0: /lon, size = 4 NC_FLOAT (Coordinate is lon)
lon attribute 0: units, size = 12 NC_CHAR, value = degrees_east

/g1/lon
lon[0]=0 degrees_east
lon[1]=90 degrees_east
lon[2]=180 degrees_east
lon[3]=270 degrees_east

Concatenate two nc4 files of different time span

3 Answers3