How to read a file from s3 using s3fs

Question

I have the following method in Python:

def read_file(self, bucket, table_name, file_name, format="csv"):
    data = None

    read_from_path = f"s3://{bucket}/{table_name}/{file_name}"

    try:
        fs = s3fs.S3FileSystem(
            key=self.dwh_s3_client_key, secret=self.dwh_s3_client_secret
        )

        with fs.open(read_from_path, "r") as f:
            data = f.read()

        return data
    except Exception as e:
        raise Exception(
           f"Failed to read file {read_from_path} from S3 due to this error:\n`{str(e)}`"
        )

The problem is that I get an Access Denied error, but I'm able to read the files in the bucket, it fails only when attempting to read the content of one of the files.

Is what I am doing the right way to read a file from s3 using s3fs?

score 0 · Answer 1 · answered Jun 09 '23 at 17:55

This looks like a permission issue to me. Could you please make sure the following Actions are added to your IAM role?

{
      "Action": [
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::<bucket-name>/*"
    }

How to read a file from s3 using s3fs

1 Answers1