Using s3fs with MinIO: FileNotFoundError when running s3fs locally, but works with boto3 and also with s3fs in Docker-compose setup

Question

I'm trying to access files in a MinIO bucket using s3fs. The code works with boto3 locally and also with s3fs when using a docker-compose setup. However, when I try to run s3fs locally, I get a FileNotFoundError.

Here's the working code using boto3:

import boto3
from io import BytesIO
access_key = 'TEST_MINIO_USER'
secret_key = 'TEST_MINIO_PASSWORD'
endpoint_url = 'http://localhost:9000'
s3 = boto3.client('s3', endpoint_url=endpoint_url, aws_access_key_id=access_key, aws_secret_access_key=secret_key)
bucket_name = 'test-bucket'
file_key = '01_raw/companies.csv'
file_content = BytesIO()
s3.download_fileobj(bucket_name, file_key, file_content)
file_content.seek(0)
print(file_content.read().decode('utf-8'))
...
14296,,Netherlands,4.0,f
...

Here's the non-working code using s3fs:

import s3fs
fs = s3fs.S3FileSystem(
     key=access_key,
     secret=secret_key,
     client_kwargs={'endpoint_url': endpoint_url}
 )
file_path = f"{bucket_name}/{file_key}"
with fs.open(file_path, 'r') as file:
     file_content = file.read()

This throws the following error:

s3fs/core.py", line 1294, in _info
    raise FileNotFoundError(path)
FileNotFoundError: test-bucket/01_raw/companies.csv

Listing bucket items also doesn't work:

>>> for obj in s3.list_objects(Bucket=bucket_name)['Contents']:
...     print(obj['Key'])

01_raw/companies.csv

>>> for file in fs.ls(bucket_name):
...     print(file)

What's interesting here is that when both my s3fs process and minio run in a docker-compose setup, then it works fine.

At the moment, I want to only run minio in a container and s3fs on my MacOS (have also reproduced on Ubuntu). It works fine with boto3, but s3fs leaves me hanging, as illustrated above.

I've also compared s3fs versions inside and outside of my container:

% venv/bin/python -m pip freeze | grep boto3
boto3==1.26.104
% venv/bin/python -m pip freeze | grep s3fs
s3fs==2023.3.0
root@921ef19d6ce6:/# pip freeze | grep s3fs
s3fs==2023.3.0

I did make an interesting discovery: As soon as I pip install boto3 within the container, also there my s3fs method stops working. Any idea how this might be happening?

And then when I downgrade again to pip install "botocore==1.27.59" it works again.

Any ideas for how to fix would be greatly appreciated. :)

Using s3fs with MinIO: FileNotFoundError when running s3fs locally, but works with boto3 and also with s3fs in Docker-compose setup

0 Answers0