0

Following the Q here xarray read remote grib file on s3 using cfgrib

How would I convert the following code to use in the backend_kwargs of xarray's open_dataset.

import fsspec
import xarray as xr

uri = "simplecache::s3://mf-nwp-models/arpege-world/v2/2021-02-16/00/UGRD/10m/0h.grib2"

file = fsspec.open_local(uri, s3={'anon': True}, filecache={'cache_storage':'/tmp/files'})

ds = xr.open_dataset(file, engine="cfgrib")

To

ds = xr.open_dataset(
    uri,
    engine="cfgrib",
    backend_kwargs=dict(s3={"anon": True}, filecache={"cache_storage": "/tmp/files"}),
)

but get ERROR:cfgrib.messages:Can't create file 'simplecache::s3://mf-nwp-models/arpege-world/v2/2021-02-16/00/UGRD/10m/0h.grib2.90c91.idx'

Ray Bell
  • 1,508
  • 4
  • 18
  • 45

1 Answers1

1

Sorry, this is not possible as things stand. xr.open_dataset expects either a URL which fsspec can turn into an open file-like object, or a local path. open_local's task is to handle a URL and return a string to a local copy - so it doesn't quite fit. fcgrib handles open file-like objects by trying to extract the local path from them and (re-)opening that.

It would be reasonable to have a function in fsspec which can do what open_local does, but hand back an open file. See discussion at fsspec-reference-maker for ideas around how we can handle the opening of (many) remote grib2s by temporarily copying them locally.

mdurant
  • 27,272
  • 5
  • 45
  • 74