R crashes while opening netcdf file

Question

I downloaded a netcdf file and trying to open it in R. Here's my code

download.file("https://data.giss.nasa.gov/impacts/agmipcf/agmerra/AgMERRA_1980_prate.nc4",destfile = "AgMERRA_1980_prate.nc4", method="libcurl")

I want to open the netcdf file using R

library(ncdf4)
my.file <- nc_open("AgMERRA_1980_prate.nc4")

However, everytime I do this, R crashes.

Is there something wrong in my code or is it something wrong with R studio?

sessionInfo() R version 3.5.0 (2018-04-23) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 7 x64 (build 7601) Service Pack 1

Matrix products: default

EDIT

If I manually download the file, I am able to open it. So there must be something wrong in the way I am downloading it. Any suggestions?

score 3 · Accepted Answer · answered Aug 06 '20 at 20:28

3

I suspect this question is duplicate of Downloading NetCDF files with R: Manually works, download.file produces error. @Luis's suggestion there of using mode = "wb" rather than the default of mode = "w" was successful in avoiding nc_open() crashes for me with R 4.0.2, RStudio 1.3.959, and ncdf 1.17. wb tells download.file() to treat the file as binary, consistent with the netCDF format.

For the data of interest here, it would be

download.file("https://data.giss.nasa.gov/impacts/agmipcf/agmerra/AgMERRA_1980_prate.nc4", destfile = "AgMERRA_1980_prate.nc4", method = "libcurl", mode = "wb")

answered Aug 06 '20 at 20:28

Todd West

326
1
8

Very useful! With `model = "wb"` it works for me with any of the methods on Windows. – Danny Jul 21 '21 at 17:44
1

Hi ! I am getting a similar problem: R crashing when trying to open .nc files automatically downloaded from a thredds server, but having no problems with manually saved files. However, even in my case, using `mode = "wb"` doesn't seem to solve the problem... Did someone find another explanation ? – Timelate Oct 25 '22 at 14:46
+1 same issue with direct downloads from thredds servers. It seems to be related to larger file sizes >1gb (smaller temporal subsets of the same spatial extent download fine) – Thomas Moore Jan 07 '23 at 04:53
One possibility, for folks not on Windows, is it's probably `mode = "ab"` that's appropriate. I haven't tried on *nix and haven't needed to download large netCDFs, so don't have direct experience here. – Todd West Jan 10 '23 at 18:02

score 0 · Answer 2 · answered Aug 18 '18 at 21:43

I am unsure what is happening here, might be windows specific. I tried to download without the argument method="libcurl" and it seems to work.

download.file("https://data.giss.nasa.gov/impacts/agmipcf/agmerra/AgMERRA_1980_prate.nc4",
destfile = "AgMERRA_1980_prate.nc4")



library(ncdf4)
my.file <- nc_open("AgMERRA_1980_prate.nc4")


File AgMERRA_1980_prate.nc4 (NC_FORMAT_NETCDF4):

     1 variables (excluding dimension variables):
        short prate[longitude,latitude,time]   (Chunking: [1440,720,1])  (Compression: level 9)
            _FillValue: 32767
            description: Precipitation Rate
            units: mm/day
            add_offset: 0
            scale_factor: 0.100000001490116
            vMin_original_data: 0
            vMax_original_data: 457.399993896484
            vRange: 457.399993896484

     3 dimensions:
        time  Size:366   *** is unlimited ***
            units: days since 1980 01-01-01 12:00:00
        latitude  Size:720
            units: degrees_north
        longitude  Size:1440
            units: degrees_east

    4 global attributes:
        history: Tue Aug 12 16:42:13 EDT 2014
        source: AgMIP / Alex Ruane
        title: AgMERRA v1.1 Precipitation Rate
        center: NASA GISS

My session info -

R version 3.5.0 (2018-04-23)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.6

R crashes while opening netcdf file

2 Answers2