Stack Overflow
Stack Overflow

Questions tagged [fsspec]

Filesystem interfaces for python

A low level IO library used by many pydata packages. See the docs https://filesystem-spec.readthedocs.io/en/latest/

39 questions
1
vote
2 answers

Unable to Save Data to HDF5 File using fsspec in Python

I'm trying to save data (3D numpy array) to an HDF5 file using fsspec in Python, but I'm encountering issues and I am unable to successfully write the data to the file. The bigger picture is I am trying to amend this dataset class to load/write…
twinsten
  • 11
  • 4
1
vote
1 answer

fsspec - is there a way how we can get paginated response from sftp?

I use fsspec which uses in-built capabilities of paramiko but could not really find a way how we can paginate the response. Is there a way to have that functionality over here? The use-case is like every directory has 100000 files and listing all of…
LearnerJS
  • 298
  • 2
  • 14
1
vote
0 answers

Pandas ExcelWriter workaround for fsspec URLs?

Is there a workaround for using pandas ExcelWriter to append to a fsspec URL? I am working out of OneDrive and need to automatically append a master xlsx file with each new xlsx file that gets uploaded to the OneDrive folder (a new xlsx file get…
rpsilberg
  • 11
  • 1
1
vote
2 answers

Streaming and caching tabular data with fsspec, parquet and Pyarrow

I’m trying to stream data from parquet files stored in Dropbox (but it could be somewhere else, S3, gdrive, etc…) and reading in Pandas, while caching it. For that I’m trying to use fsspec for Python Following these instructions that’s what I’m…
Luiz Tauffer
  • 463
  • 6
  • 17
1
vote
2 answers

Paramiko SFTP file renaming: OSError('Extended request not supported.')

I can't find anything about this exception. I am trying to rename a remote file on a local (Windows) SFTP server with fsspec. Paramiko behind the scenes is doing a posix_rename(). What does the error mean? fs.rename(old_file_path,…
cdonner
  • 37,019
  • 22
  • 105
  • 153
1
vote
2 answers

What is the working combination of the s3fs and fsspec version? ImportError: cannot import name 'maybe_sync' from 'fsspec.asyn'

I am using the latest version of s3fs-0.5.2 and fsspec-0.9.0, when import s3fs, encountered the following error: File "/User/.conda/envs/py376/lib/python3.7/site-packages/s3fs/__init__.py", line 1, in from .core import S3FileSystem, S3File …
xsqian
  • 199
  • 5
  • 13
1
vote
0 answers

Unable to read a url using fsspec https filesystem implementation directly

I am using fsspec package to implement a function to read an https file. _hostname = socket.gethostname() proxy_auth = aiohttp.BasicAuth(_hostname, pwd) of = fsspec.filesystem("https", client_kwargs={"trust_env":True, "auth":proxy_auth}) http_urls =…
MSS
  • 3,306
  • 1
  • 19
  • 50
1
vote
1 answer

Import error when using pandas for fsspec in python?

I had an issue where pandas would report an import error, upon attempting to use a buffer from io's StringIO class, during the use of read_csv() from pandas.
user1802263
  • 93
  • 1
  • 7
1
vote
1 answer

xarray read remote grib file on s3 using cfgrib

Can the crgrib engine handle reading remote files? It doesn't look like it according to Martin Durant's comment (https://github.com/ecmwf/cfgrib/issues/198#issuecomment-772852412) There is a smallish grib file hosted on s3:…
Ray Bell
  • 1,508
  • 4
  • 18
  • 45
1
vote
1 answer

getting KeyError '.zmetadata' when opening remote zarr store

Trying to read in a zarr store from s3 using xarray. Getting a Key Error. Any thoughts import fsspec import xarray as xr uri = "s3://era5-pds/zarr/2020/12/data/eastward_wind_at_10_metres.zarr" ds = xr.open_zarr(fsspec.get_mapper(uri, anon=True),…
Ray Bell
  • 1,508
  • 4
  • 18
  • 45
1
vote
0 answers

Can Python package fsspec read SSH config?

I would like to access remote SSH server files within Python, and found fsspec. However, there seems to be few code usage examples. In particular, I can connect by specifying all SSH config options in the function as: fsspec.filesystem('sftp',…
Claire
  • 639
  • 9
  • 25
0
votes
0 answers

How to write into multiple excel sheets using fsspec?

I am trying to write several data frames into multiple sheets using fsspec in azure synapse, but the issue is that the data getting overwrite to single sheet and finally only 1 data frame is availabel in the output file. How can I correct this to…
suwaris
  • 1
0
votes
1 answer

FileNotFoundError using gcsfs and pandas, but only on my machine

I have used pandas and gcsfs regularly in the past. Recently, I have been getting errors when trying to do so. I cannot reproduce the error on other systems. On my system, I get the error using both python 3.9.16 and 3.11.1. As you can see from the…
seandavi
  • 2,818
  • 4
  • 25
  • 52
0
votes
1 answer

fsspec - way to use proxy to connect to sftp in Python

authentication_kwargs = dict() authentication_kwargs["password"] = password sftp = fsspec.filesystem("sftp", host=host, port=port, username=username,**authentication_kwargs) This is how I connect to sftp using host, port, username and password. How…
LearnerJS
  • 298
  • 2
  • 14
0
votes
1 answer

Use connection pooling with python sshfs (fsspec) in Python

I'm using sshfs to fetch video files from a remote SSH storage: @app.get("/video/{filename}") async def video_endpoint( filename, range: str = Header(None), db=Depends(get_db) ): # pylint: disable=redefined-builtin """ Endpoint for…
Daniel
  • 1,398
  • 4
  • 20
  • 47
1
2
3