0

I'm trying to open thousands of netcdf files and pull the data into a dataframe that I can save as a csv file.

So after putting all the ncdf files names in a csv file as a list, I ended up using this code to create a dataframe populated with the data from the ncdf files.

vec0 <- vector()
time <- c(vec0,1:28413)

vec1 <- vector()
temp <- c(vec1,1:28413)

vec2 <- vector()
sphum <- c(vec2,1:28413)

vec3 <- vector() 
rain <- c(vec3,1:28413)

vec4 <- vector()
totprcp <- c(vec4,1:28413)

for (i in 1:length(filenames))
  {ncdata=nc_open(filenames[i]) 

 nctime=ncvar_get(ncdata,"time") 
  time[i] = nctime[1]

nctemp=ncvar_get(ncdata,"Tair_f_inst") 
  temp[i] = nctemp[1]

 nchum=ncvar_get(ncdata,"Qair_f_inst")
  sphum[i] = nchum[1]

  ncrain=ncvar_get(ncdata,"Rainf_tavg") 
  rain[i] = ncrain[1]

  ncprcp=ncvar_get(ncdata,"Rainf_f_tavg")
  totprcp[i] = ncprcp[1]
  nc_close(filenames[i])}

  cbind(time,temp,sphum,rain,totprcp)

But I'm only getting accurate data for the first row and its filling in a sequence of numbers (1 through 28412) for the rest of the rows. I think my mistake lies in the way I wrote the nc_close component of the code. Any ideas?

Ktass
  • 47
  • 1
  • 1
  • 7
  • I'm really new to using R so my code may have some redundant components. The for-loop is necessary because i'm trying to open 28000 ncdf files and extract the data from each one and put it into this dataframe I've created with the variables specified. I'm not sure how else to do this without a for-loop. The code (without nc_close) works fine when I run it on 3 files, but when i run it for a larger number R crashes and gives me an error saying too many files open, so I added the nc_close component into the for-loop so that it would open the ncdf file, extract, close the file, and repeat. – Ktass Jan 22 '19 at 19:56

1 Answers1

0

nc_close() should be applied to the nc object itself, not to the filename. I.e. to "An object of class ncdf4 (as returned by either function nc_open or function nc_create."

So, you should use

nc_close(ncdata)
dww
  • 30,425
  • 5
  • 68
  • 111
  • I'm getting this error when I use your suggested simplified code: Error in ncvar_get_inner(ncid2use, varid2use, nc$var[[li]]$missval, addOffset, : Error: variable has 3 dims, but count has 1 entries. They must match! How do I fix this? – Ktass Jan 22 '19 at 23:21
  • `count` must have same length as variable dimensions. If, say, `Rainf_tavg` is 3d, then you could use `ncvar_get(ncdata, "Rainf_tavg", count=c(1,1,1)`, or `ncvar_get(ncdata, "Rainf_tavg")[1]`. But it all depends exactly what you want to extract. If these are spatial variables, are you sure you want only a single value? This would be a value at only a single specific location. This is why it's important to provide a reproducible example, including data. It is impossible to predict things like this unless you can provide the data and a clear description of your needs. – dww Jan 23 '19 at 04:03