We're building a python abstraction on top of AWS S3 and MinIO, and we're using the s3fs API to transparently talk to either. While both the boto and the minio APIs allow us to retrieve all the versions of a given file (object), we can't find a way to do the same via the s3fs API. The API documentation mentions a version_aware argument described as:
Whether to support bucket versioning...When set to True, filesystem instances will use the S3 ListObjectVersions API call to list directory contents, which requires listing all historical object versions.
However, when we use the ls method on a directory that contains one object with three versions, we only get back the latest version - while the minio API would return all of them.
from s3fs import S3FileSystem
client = S3FileSystem(
key="...",
secret="...",
endpoint_url="https://minio-dev.xxx.com",
use_ssl=False,
version_aware=True
)
objects = client.ls("gabe-tests-1/mher", detail=True)
print(f"Retrieved {len(objects)} objects")
for o in objects:
print(f" {objects}")
Is there anything we've overlooked?
