1

Are there any workarounds to the problem with ncdf4::nc_open not being able to access some .nc files from a URL? I would like to avoid having to download the file first since this is for a shiny app deployed on a server and so I want to avoid users being able to download files to the server.

Some URLs work e.g. this OPeNDAP URL from a THREDDS server:

library(ncdf4)
nc <- nc_open("https://dapds00.nci.org.au/thredds/dodsC/uc0/Test_pixel_count.nc")

But others do not e.g. this NetCDF Subset Service URL from a THREDDS server:

nc <- nc_open("https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1")
# Error in nc_open("https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1") : 
# Error in nc_open trying to open file https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1

or this file directly from a website:

nc <- nc_open("https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc")
# syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
# context: <!DOCTYPE^ HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"><html><head><title>404 Not Found</title></head><body><h1>Not Found</h1><p>The requested URL was not found on this server.</p></body></html>
# Error in R_nc4_open: NetCDF: file not found
# Error in nc_open("https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc") : 
# Error in nc_open trying to open file https://www.unidata.ucar.edu/software/netcdf/examples/ECMWF_ERA-40_subset.nc

What could be the reason why some methods work and some don't, and are there any ways to fix this?

Danny
  • 448
  • 3
  • 15

1 Answers1

4

I ran into the same issue and contacted the creator of ncdf4. Here is his reply:

The ncdf4 R package is an interface to the underlying netcdf library on your machine. The R package cannot avoid limitations or problems with the netcdf library itself. You can always see how the underlying netcdf library handles a URL by just trying to use "ncdump" with the URL. For example,

% ncdump "https://dapds00.nci.org.au/thredds/dodsC/uc0/Test_pixel_count.nc"

Gives:

netcdf Test_pixel_count {
dimensions:
    lat = 283 ;
    lon = 451 ;
variables:
    byte crs ;
            crs:_Unsigned = "false" ;
            crs:grid_mapping_name = "latitude_longitude" ;
            crs:long_name = "CRS definition" ;
            crs:longitude_of_prime_meridian = 0. ;
            crs:semi_major_axis = 6378137. ;
            crs:inverse_flattening = 298.257222101 ;
            crs:spatial_ref = "GEOGCS[\"GEOCENTRIC DATUM of AUSTRALIA\",DATUM[\"GDA94\",SPHEROID[\"GRS80\",6378137,298.257222101]],PRIMEM[\"Greenwich\",0],UNIT[\"degree\",0.0174532925199433],AXIS[\"Longitude\",EAST],AXIS[\"Latitude\",NORTH]]" ;
            crs:GeoTransform = "141.4607205456306 0.0075 0 -22.95189554231917 0 -0.0075 " ;

etc. I.e., the netcdf library itself can read this URL/file.

However when I do this:

%  ncdump "https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1"

I get this error:

ncdump "https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1"
ncdump: https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1: https://dapds00.nci.org.au/thredds/ncss/uc0/Test_pixel_count.nc?var=Band1&north=-22.9556&west=142&east=143&south=-25.0706&disableProjSubset=on&horizStride=1: 
NetCDF: Malformed or unexpected Constraint

So the netcdf library itself cannot access that URL/file, and therefore ncdf4 cannot do so. There's not anything I would be able to do to the ncdf4 library to make up for this.

You might try contacting the netcdf library folks and ask if this is expected behavior or a bug.

chm
  • 51
  • 6
  • Thanks a lot, this is really useful information! – Danny Dec 16 '21 at 18:37
  • Hey @Danny, @chm; were you able to solve this issue, I am trying to access a .nc file through ncdf4. However, no success! Please let me know if you have any update on this issue. – Prasad Thota Mar 10 '23 at 03:49
  • I haven't re-checked recently, but as @chm mentioned, it seems to be a limitation of the netcdf library. I found that URLs sometimes worked, but OPeNDAP URL from a THREDDS servers generally didn't always work. It may be related to how the server is set up but I'm really not sure. – Danny Mar 13 '23 at 08:53