0

I want to create s3 file system for uploading files to s3 bucket using the pyarrows write_to_dataset function

fs = s3fs.S3FileSystem()
pa.parquet.write_to_dataset(table, root_path=output_folder, filesystem=fs,
                            compression='snappy', partition_filename_cb=lambda x: filename)

but I got to know that s3fs library is still in the beta phase.

is there any production/stable library available for creating a s3 file system?

if not is there other way like boto3 to create s3 file system?

Andrew Gaul
  • 2,296
  • 1
  • 12
  • 19
amber_coder
  • 47
  • 4
  • Welcome to Stack Overflow. Where did you read that S3FS is in beta? There's already been 18 releases and the latest release is currently 1.91, far from beta. Is there a reason you are trying to find alternatives? S3FS is production ready. – Ermiya Eskandary Dec 12 '22 at 13:39
  • here Development Status = beta on https://pypi.org/project/s3fs/ – amber_coder Dec 12 '22 at 14:32
  • or maybe i am refering to wrong library here version mentioned as 2022.11.0 – amber_coder Dec 12 '22 at 14:33
  • Ah OK - I understand. Try reaching out to Martin Durant, the maintainer (mdurant@continuum.io) and ask their advice on using S3FS within production. Most likely, it's still in beta because of the lack of enterprise support. – Ermiya Eskandary Dec 12 '22 at 16:19
  • Note the difference between s3fs fuse and s3fs-python. – Andrew Gaul Dec 12 '22 at 21:00

0 Answers0