For questions related to the Python s3fs library
Use this tag for questions related to the Python s3fs library.
Not to be confused with the s3fs tag, which is for mounting an s3fs bucket on a local mount point and has nothing to do with Python.
Questions tagged [python-s3fs]
85 questions
0
votes
0 answers
Writing to an S3 key based on current date
I am trying to write a csv in S3 using S3FileSystem in python. Every time I write, I create a file with the current date-time ('%Y-%m-%d-%H-%M-%S') within a key of the current date ('%Y-%m-%d') so that the data is organized per date. I was thinking…
nad
- 2,640
- 11
- 55
- 96
0
votes
1 answer
What is the correct way to set timeouts in s3fs.S3FileSystem?
I've tried various ways to set the read timeout on a s3fs.S3FileSystem object such as
s3 = s3fs.S3FileSystem(s3_additional_kwargs={"read_timeout": 500}, config_kwargs={"read_timeout": 500} )
or
s3.read_timeout = 500
But none of them seem to be…
Andy
- 13
- 3
0
votes
0 answers
Dask list index out of range reading JSON files from S3
I'm trying to read a bunch of JSON files stored on S3, but is raising a list index out of range when I compute the DataFrame
My call to open the JSON files is like this:
pets_data = dd.read_json("s3://my-bucket/pets/*.json", meta=meta,…
Carlos Rojas
- 334
- 5
- 13
0
votes
1 answer
When is next release for s3fs planned
I have tried using 0.4.2 s3fs pip module, moving greater than 5GB files are failing with write fail errors.
There is an issue with 0.4.2 version of s3fs pip module, which got fixed in master. When is next release planned so that i can use this…
0
votes
1 answer
How to diagnose inconsistent S3 permission errors
I'm running a Python script in an AWS Lambda function. It is triggered by SQS messages that tell the script certain objects to load from an S3 bucket for further processing.
The permissions seem to be set up correctly, with a bucket policy that…
penguinrob
- 1,431
- 3
- 17
- 39
0
votes
1 answer
Problem uploading an sklearn model to S3 bucket using s3fs
I am trying to upload an SVR model (created with sklearn) to S3 bucket using s3fs, but I get an error saying "TypeError: a bytes-like object is required, not 'SVR'". Can anyone suggest how to transform SVR into the right format?
My code is
model =…
susannochka
- 11
- 1
0
votes
2 answers
Is there any python module through which I can mount a S3 bucket into a filesystem?
s3fs-FUSE is a CLI tool which can do this
But s3fs library does not actually mount it to the filesystem, instead it gives us apis to access the S3 buckets as file system
I'm looking for some python library which I can use in my python script to…
Deepak Sundar
- 1
- 1
0
votes
0 answers
How do I serve static file if I have mounted s3 bucket using s3fs?
I have mounted one of my buckets to my server, I am able to access and write files to it, but now I wish to play files. I have written small api that returns static file from my mounted path of s3 bucket in my server
def get_file(path):
if not…
Varun Singh
- 444
- 6
- 21
0
votes
1 answer
Does dask.s3fs actually use the cache?
When you call S3FileSystem().open(..., fill_cache=True) it looks like it still goes to S3, still does the ssl handshake etc. Is this the intended behaviour that it does not cache?
mathtick
- 6,487
- 13
- 56
- 101
-1
votes
1 answer
How do I download a linked pdf file from its url using python?
I have multiple URL's like 'https://static.nseindia.com/s3fs-public/2022-09/ind_prs01092022.pdf' and I want to loop through an array of these and download them to a local folder.
I saw that I may need to use s3fs, but I am unsure what the bucket…
wanderingtrader
- 3
- 1