0

I would like to access climate (CORDEX) data through ESGF's OPeNDAP service via Python (xarray looks like a great option).

This dataset is an example of one I cannot access: https://esg-dn1.nsc.liu.se/thredds/dodsC/esg_dataroot7/cordexdata/cordex/output/CAS-44/BOUN/MOHC-HadGEM2-ES/rcp45/r1i1p1/BOUN-RegCM4-3/v5/day/tas/v2019116/tas_CAS-44_MOHC-HadGEM2-ES_rcp45_r1i1p1_BOUN-RegCM4-3_v5_day_20060201-20101230.nc.html

I have saved credentials in a credentials.pem file, and I have a .dodsrc file in my home directory, which referms to my credentials.pem file, like so:

HTTP.VERBOSE=0
HTTP.COOKIEJAR=C:\\Users\\tsh371/.esg.dods_cookies
HTTP.SSL.VALIDATE=0
HTTP.SSL.CERTIFICATE=C:\\Users\\tsh371/.esg\\credentials.pem
HTTP.SSL.KEY=C:\\Users\\tsh371/.esg\\credentials.pem
HTTP.SSL.CAPATH=C:\\Users\\tsh371/.esg\\ca-trustroots

But when accessing the dataset, for example through xarray.open_dataset() on the dataset above, I get. [Errno -77] NetCDF: Access failure

I have tried to verify my credentials.pem is valid by accessing a tiny slice of the data. For example with

url = 'https://esg-dn1.nsc.liu.se/thredds/dodsC/esg_dataroot7/cordexdata/cordex/output/CAS-44/BOUN/MOHC-HadGEM2-ES/rcp45/r1i1p1/BOUN-RegCM4-3/v5/day/tas/v2019116/tas_CAS-44_MOHC-HadGEM2-ES_rcp45_r1i1p1_BOUN-RegCM4-3_v5_day_20060201-20101230.nc.ascii?tas%5B0:1:0%5D%5B0:1:0%5D%5B0:1:0%5D'

Then getting the data while explicitly pointing at the credentials like this, gets me the data:

import requests
requests.get(url, cert = 'C:/Users/tsh371/.esg/credentials.pem')

But while running the same request without including the cert argument, gives response 401. Note that with other climate data (fx https://dap.ceda.ac.uk/), after correctly setting up .dodsrc, I don't have to include a cert argument.

Is there something wrong with my .dodsrc file? Or is there some way to tell xarray where to look for the certificate?

Reproducible code that generates the error:

import pyesgf

import xarray as xa
import os

os.environ['HOME'] = os.environ['USERPROFILE'] # pyesgf refers to HOME, which should be equivalent to USERPROFILE on Windows
from pyesgf.logon import LogonManager

lm = LogonManager()

lm.logon(username = None, password = None, hostname = 'esgf-node.ipsl.upmc.fr', bootstrap=True)

opendapurl = 'https://esg-dn1.nsc.liu.se/thredds/dodsC/esg_dataroot7/cordexdata/cordex/output/CAS-44/BOUN/MOHC-HadGEM2-ES/rcp45/r1i1p1/BOUN-RegCM4-3/v5/day/tas/v2019116/tas_CAS-44_MOHC-HadGEM2-ES_rcp45_r1i1p1_BOUN-RegCM4-3_v5_day_20060201-20101230.nc'

xa.open_dataset(opendapurl)
TiemvdDeure
  • 66
  • 4
  • You might want to try the ESGF python client https://esgf-pyclient.readthedocs.io/en/latest/ – Robert Wilson Feb 03 '23 at 16:59
  • I used the ESGF python client to generate the credentials file. Using their example notebooks I run into the same problem. – TiemvdDeure Feb 03 '23 at 19:26
  • Can you show us the actual code you’re using which generates the error? I don’t see any xarray in here… – Michael Delgado Feb 03 '23 at 19:39
  • As @MichaelDelgado please make the problem reproducible. All people can do right now is second guess what is causing the problem – Robert Wilson Feb 03 '23 at 21:28
  • I have edited some code in that reproduces the error for me. – TiemvdDeure Feb 04 '23 at 09:50
  • Does [this](https://github.com/ESGF/esgf-pyclient/issues/50) help in any way? [This](https://github.com/ESGF/esgf-pyclient/issues/52) seems to be a related issue (note that it's the same user in both GitHub issues). – FiddlingAway Feb 05 '23 at 09:41
  • It does look like the same issue, but the response doesn't solve my problem. I tried running the examples given there, but still get an error. I'm thinking this might have something to do with either different operating systems, or the placement of the .dodsrc file? – TiemvdDeure Feb 06 '23 at 08:42

0 Answers0