Replace values in netCDF file using python

Question

I would like to replace all the -999.0 values in the foll. netCDF file by 0.0. The file is here: https://www.dropbox.com/s/ohh7tntzm2pj1uw/all_SSP3_BaU_NoCC.nc?dl=0

here's my function:

def modify_nc_value(path_inp, name_var, val_to_modify, new_value):
    """
    Replace value in netCDF vriable from val_to_modify to new_value
    Args:
        path_inp:
        name_var: Name of variable for which to modify value, if -1, then change all variables
        val_to_modify:
        new_value:

    Returns:

    """
    hndl_inp = netCDF4.Dataset(path_inp, 'r+')

    if name_var == -1:
        for var, varin in hndl_inp.variables.iteritems():
            hndl_inp[var][:][hndl_inp[var][:] == val_to_modify] = new_value
    else:
        hndl_inp[name_var][:][hndl_inp[name_var][:] == val_to_modify] = new_value

    hndl_inp.close()  # Write variable back to disk

However, the function does not seem to work. None of the -999.0 values are replaced. How to fix this?

Are the variables always 1D? I suspect not, but your code always assigns `var` to be 1D, i.e `[var][:]`. — N1B4, Jul 08 '16 at 21:18
Don't you need to sync as mentioned in http://unidata.github.io/netcdf4-python/#netCDF4.Dataset.sync? — Felippe Raposo, Jul 12 '16 at 19:34
I think for the 3D array, multidimensional indexing is not allowed by NetCDF (see [source code here](https://github.com/Unidata/netcdf4-python/blob/master/netCDF4/utils.py#L199)), but that's what you need really. The only variable in your sample file that has -999 values appears to be `c4ann` - is that what you expect? I notice that some others (e.g. `c3ann`) have a mask, where the fill value is -999. Maybe you can give more context and we can help find a way to get what you want. — J Richard Snape, Jul 13 '16 at 09:47

score 3 · Accepted Answer · answered Jul 15 '16 at 02:07

You opened the file in append mode (r+), which will only allow new values to be added. There really is no read/write mode because data sets grow huge quickly. You have two options:

Read the dataset, open a new file (mode is w, clobber is True), write to it with modifications, copy it back over the original dataset.
If the dataset is large, meaning > 5T or so, you really want to modify your code to look for -999.0 values instead.
There may be a way to do it. I've never made these hints, but you may have better luck.

Good luck! If you make it work, add it to these answers.

score 1 · Answer 2 · answered Apr 29 '20 at 08:05

If you are happy to consider doing this task from the command line, it is simple to achieve with climate data operators (CDO)

You can define a mask that has 0 where the original field has -999 and then multiple this mask with the original field.

cdo nec,-999.0 file.nc mask.nc
cdo mul file.nc mask.nc new_file.nc

or piped together

cdo mul file.nc -nec,-999.0 file.nc new_file.nc

However, I'm assuming that -999.0 is actually the "missing value" for the field identified in the metadata. In that case you can also achieve this using

cdo setmissval,0 file.nc new_file.nc

The file contents of the variable are identical both ways, but the difference is that the latter command will also change the metadata to identify zero as the "missing value". This means that 0 is correctly identified as missing in software such as ncview. However, note that any incidence of zero is now missing, including those that were already there in the original file!

Replace values in netCDF file using python

2 Answers2