Fortran output large array to netCDF nf90_enddef

Question

Hi first question on stackoverflow.

I've been stuck on this for 5 days. I want to write a netcdf file in Fortran.

I'm using netcdf/3.6.3 I am trying to output a large 43000x 18000 array named frech and some smaller ones (1d arrays).

a sample of my code is below it is a really large file and the problem is not in putting in values for the variables the problem is ending the definition of variables:

  print*,"nunks is",nunks
  print*,"neqns is",neqns


  ok=nf90_create('michalek.nc', NF90_CLOBBER, ncid)
  print *,"create ok=",ok

  ok=   nf90_def_dim(ncid,"nunks", nunks, nunks_dimid)
  print *,"def nunks dimension ",ok
  ok=   nf90_def_dim(ncid,"neqns", neqns, neqns_dimid)
  print *,"def neqns dimension ",ok
  dimids=(/neqns_dimid, nunks_dimid/)
  print *,dimids
  ok=   nf90_def_var(ncid,"frech", NF90_REAL, dimids, frech_varid)
  print *,"def frech",ok
  ok=   nf90_def_var(ncid,"src", NF90_REAL, nunks_dimid, src_varid)
  print *,"def src",ok
  ok=   nf90_def_var(ncid,"csrc", NF90_REAL, nunks_dimid, csrc_varid)
  print *,"define csrc",ok
  ok=   nf90_def_var(ncid,"dat", NF90_REAL, neqns_dimid, dat_varid)
  print *,"define dat",ok
  ok=   nf90_def_var(ncid,"cdat", NF90_REAL, neqns_dimid, cdat_varid)
  print *,"define cdat",ok
  ok=   nf90_enddef(ncid)
  print *,"end dif ", ok
  ok=   nf90_put_var(ncid, frech_varid, frech)
  print *, 'frech put in ok=',ok
  ok=   nf90_put_var(ncid, src_varid, src)
  print *, 'src put in ok=',ok
  ok=   nf90_put_var(ncid, csrc_varid, csrc)
  print *, 'csrc put in ok=',ok
  ok=   nf90_put_var(ncid, dat_varid, dat)
  print *, 'dat put in ok=',ok
  ok=   nf90_put_var(ncid, cdat_varid, cdat)
  print *, 'cdat put in ok=',ok
  ok=   nf90_close(ncid)
  print *, 'close?',ok

I understand that ok=0 when the file is correctly read However when I get to the stage of ending the defintions of the file (nf90_enddif) ok is returned as =-62 and the netcdf file is not created. I imagine its a problem with a too large array but I can't fix this problem

the relevant output for the above code is:

nunks is       43894
neqns is       18144
create ok=           0
def nunks dimension            0
def neqns dimension            0
     2           1
def frech           0
def src           0
define csrc           0
define dat           0
define cdat           0
end dif          -62
frech put in ok=         -39
src put in ok=         -39
csrc put in ok=         -39
dat put in ok=         -39
cdat put in ok=         -39
close?         -62

Thanks for any help!

Regards Peter :)

Did you play with the variable size to see if you success with smaller size? For example go for 1000x1000 and increase gradually to see where you get the problem — innoSPG, Jul 31 '15 at 01:01
That's a good idea. But I'd like to avoid experimenting too much on the code because it takes 10 minutes to execute. Maybe I should try and shorten aspects of the code. — Weatherman, Aug 01 '15 at 12:10

score 1 · Accepted Answer · edited Jun 20 '20 at 09:12

It is possibly due to the limitation in the size of variables in netcdf, 4 giga (see below). You might want to slice your array before saving it or move to netcdf-4/hdf-5.

Have all netCDF size limits been eliminated?

The netCDF-4 HDF5-based format has no practical limits on the size of a variable.

However, for the classic and 64-bit offset formats there are still limits on sizes of netCDF objects. Each fixed-size variable (except the last, when there are no record variables) and the data for one record's worth of a single record variable (except the last) are limited in size to a little less that 4 GiB, which is twice the size limit in versions earlier than netCDF 3.6.

The maximum number of records remains 232-1.

For details, see http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#Large%20File%20Support10

Fantastic! I had no idea that changing from netcdf 4 and 3 could make such a difference. It worked when I loaded a newer version of netCDF... If only I had asked 5 days ago ;) thxs for your help — Weatherman, Aug 01 '15 at 12:13
I am glad to help. Do not forget to accept the answer, so that people can stop digging for other solution. — innoSPG, Aug 01 '15 at 18:44

score 0 · Answer 2 · answered Jul 31 '15 at 03:11

In NetCDF classic format when not using unlimited dimensions you are only allowed 1 variable that exceeds 2GiB. It also must be the last variable in the dataset and the offset to the beginning be no more than 2GiB. It looks like moving frech to the end should work.

score 0 · Answer 3 · answered Jul 31 '15 at 21:03

In Argonne-Northwestern's Parallel-NetCDF (http://cucis.ece.northwestern.edu/projects/PnetCDF/ ) we relaxed the file size limits by creating the CDF-5 file format. It took a while, but I think we're making progress on getting Unidata-NetCDF to recognize it.

If you control the readers and writers of your data, you might want to see if parallel-netcdf with the newer CDF-5 file format works for you. If you need to collaborate with others, might want to wait until it's officially part of Unidata's NetCDF (and let Unidata know it's useful to you!).

Fortran output large array to netCDF nf90_enddef

3 Answers3