pack/compress netcdf data ("add offset" and "scale factor") with CDO, NCO or similar

Question

I have heavy netCDF files with floating 64-bits precision. I would like to pack using specific values for the add_offset and scale_factor parameters (so then I could transform to short I16 precision). I have found information for unpacking with CDO operators but not for packing.

Any help? Thank you in advance!

Edit:

diego@LAcompu:~/new$ ncks -m in.nc
netcdf in {
  dimensions:
    bnds = 2 ;
    lat = 202 ;
    lon = 62 ;
    time = UNLIMITED ; // (15777 currently)

  variables:
    float lat(lat) ;
      lat:standard_name = "latitude" ;
      lat:long_name = "latitude" ;
      lat:units = "degrees_north" ;
      lat:axis = "Y" ;

    float lon(lon) ;
      lon:standard_name = "longitude" ;
      lon:long_name = "longitude" ;
      lon:units = "degrees_east" ;
      lon:axis = "X" ;

    double t2m(time,lat,lon) ;
      t2m:long_name = "2 metre temperature" ;
      t2m:units = "Celsius" ;
      t2m:_FillValue = -32767. ;
      t2m:missing_value = -32767. ;

    double time(time) ;
      time:standard_name = "time" ;
      time:long_name = "time" ;
      time:bounds = "time_bnds" ;
      time:units = "hours since 1900-01-01 00:00:00.0" ;
      time:calendar = "gregorian" ;
      time:axis = "T" ;

    double time_bnds(time,bnds) ;
} // group /
diego@LAcompu:~/new$ ncap2 -v -O -s 't2m=pack_short(t2m,0.00166667,0.0);' in.nc out.nc
ncap2: WARNING pack_short(): Function has been called with more than one argument
diego@LAcompu:~/new$ ncks -m out.nc
netcdf out {
  dimensions:
    lat = 202 ;
    lon = 62 ;
    time = UNLIMITED ; // (15777 currently)

  variables:
    float lat(lat) ;
      lat:standard_name = "latitude" ;
      lat:long_name = "latitude" ;
      lat:units = "degrees_north" ;
      lat:axis = "Y" ;

    float lon(lon) ;
      lon:standard_name = "longitude" ;
      lon:long_name = "longitude" ;
      lon:units = "degrees_east" ;
      lon:axis = "X" ;

    short t2m(time,lat,lon) ;
      t2m:scale_factor = -0.000784701646794361 ;
      t2m:add_offset = -1.01787074416207 ;
      t2m:_FillValue = -32767s ;
      t2m:long_name = "2 metre temperature" ;
      t2m:missing_value = -32767. ;
      t2m:units = "Celsius" ;

    double time(time) ;
      time:standard_name = "time" ;
      time:long_name = "time" ;
      time:bounds = "time_bnds" ;
      time:units = "hours since 1900-01-01 00:00:00.0" ;
      time:calendar = "gregorian" ;
      time:axis = "T" ;
} // group /

Charlie Zender · Accepted Answer · 2022-03-26T02:16:27.283

3

NCO will automatically pack with optimal values for scale_factor and add_offset with, e.g.,

ncpdq -P in.nc out.nc

and you can add lossless compression as well with

ncpdq -P -L 1 -7 in.nc out.nc

Documentation at http://nco.sf.net/nco.html#ncpdq

and ncap2 accepts specific values of scale_factor and add_offset for per-variable packing with pack() documented at http://nco.sf.net/nco.html#ncap_mth

Demonstration:

zender@spectral:~$ ncap2 -v -O -s 'rec_pck=pack(three_dmn_rec_var,-0.001,40.0);' ~/nco/data/in.nc ~/foo.nc
zender@spectral:~$ ncks -m ~/foo.nc
netcdf foo {
  dimensions:
    lat = 2 ;
    lon = 4 ;
    time = UNLIMITED ; // (10 currently)

  variables:
    float lat(lat) ;
      lat:long_name = "Latitude (typically midpoints)" ;
      lat:units = "degrees_north" ;
      lat:bounds = "lat_bnd" ;

    float lon(lon) ;
      lon:long_name = "Longitude (typically midpoints)" ;
      lon:units = "degrees_east" ;

    short rec_pck(time,lat,lon) ;
      rec_pck:scale_factor = -0.001f ;
      rec_pck:add_offset = 40.f ;
      rec_pck:_FillValue = -99s ;
      rec_pck:long_name = "three dimensional record variable" ;
      rec_pck:units = "watt meter-2" ;

    double time(time) ;
      time:long_name = "time" ;
      time:units = "days since 1964-03-12 12:09:00 -9:00" ;
      time:calendar = "gregorian" ;
      time:bounds = "time_bnds" ;
      time:climatology = "climatology_bounds" ;
} // group /

edited Mar 26 '22 at 02:16

answered Mar 25 '22 at 22:43

Charlie Zender

5,929
14
19

Thank you very much! I tried this but results are different to the offset and scale parameters I fixed. For example, I use `ncap2 -v -O -s 't2m=pack_short(t2m,0.00166666666666667,0.0);' in.nc out.nc` but results indicate `scale_factor: -0.000784701646794361` and `add_offset: -1.01787074416207` – Diego Hernandez Mar 26 '22 at 01:58
1

It works as advertised for me see above. – Charlie Zender Mar 26 '22 at 02:15
thank you! I edited the post with my code – Diego Hernandez Mar 26 '22 at 03:02
Then use pack() not pack_short(), it's the only difference between our commands – Charlie Zender Mar 27 '22 at 14:10
Yes I tried also, with same results for `pack_short`. And also I get this warning `WARNING pack(): Function has been called with more than one argument` that you don't obtain (?) – Diego Hernandez Mar 27 '22 at 16:30
1

That is correct. I do not understand why our results differ. Be sure you are using a recent NCO... – Charlie Zender Mar 28 '22 at 17:07
now it works and I didn't get the warning... thank you very much! it was the version I suppose... – Diego Hernandez Mar 28 '22 at 21:10

ClimateUnboxed · Answer 2 · 2022-06-23T17:19:51.780

0

So this was simpler than I thought in cdo

cdo pack in.nc out.nc

This packs with optimal add_offset and scale_factor, converting the field to I16.

edited Jun 23 '22 at 17:19

answered Mar 25 '22 at 20:02

ClimateUnboxed

7,106
3
41
86

Thank you very much! Actually I have other referential datasets with ncdf4 format, short I16, and specific values of offset and scale parameters, so I'm trying to get that specific format, starting from ERA5 downloaded data and then modifying – Diego Hernandez Mar 25 '22 at 22:06
ps: sorry for making your comment outdated, I deleted the part of the answer related to compression. – ClimateUnboxed Jun 23 '22 at 17:21

pack/compress netcdf data ("add offset" and "scale factor") with CDO, NCO or similar

2 Answers2

Linked