0

I have a s3 structure like s3://my-bucket/data/2020/03/23/01/data.csv I want to check if s3://my-bucket/data/2020/03/23 exist.

I can check if the CSV file exist but I cant use that because file name might change so I want to check if the folder exists.

franklinsijo
  • 17,784
  • 4
  • 45
  • 63
Aditi Kishore
  • 7
  • 1
  • 5
  • Check [here](https://stackoverflow.com/questions/9551347/check-if-a-key-with-a-certain-prefix-exists-in-amazon-s3-bucket). – Marcin Mar 23 '20 at 11:25
  • _Why_ do you wish to check if it exists? If you are just checking if a file of any name exists, then list the bucket with the given Prefix. – John Rotenstein Mar 23 '20 at 22:58

2 Answers2

1

This might not be possible, depending what you are expecting.

First, it's worth mentioning that folders do not actually exist in Amazon S3.

For example, you could run this command to copy a file to S3:

aws s3 cp foo.txt s3://my-bucket/data/2020/03/23/

This would put the file in the data/2020/03/23/ path and those four directories would "appear" in the console, but they don't actually exist. Rather, the Key (filename) of the object contains the full path.

If you were then to delete the object:

aws s3 rm s3://my-bucket/data/2020/03/23/foo.txt

then the four directories would "disappear" (because they didn't exist anyway).

It is possible to cheat by clicking "Create Folder" in the S3 management console. This will create a zero-length object with the name of the folder (actually, with the name of the full path). This causes the directory to appear in the bucket listing, but that is purely because an object exists in that path.

Within S3, directories are referred to as CommonPrefixes and commands can be used that reference a prefix, rather than referencing a directory.

So, you could list the bucket, providing the path as a prefix. This would then return a list of any objects that are within that path.

However, the best answer is: Just pretend that it exists and everything will work fine.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
0

What I would do is what John Rotenstein mentioned in a comment: list the contents of a bucket specifying a prefix, which in this case is the path to the directory you are interested in (data/2020/03/23/01).

import boto3
from botocore.exceptions import ClientError

def folder_exists(bucket_name, path_to_folder):
    try:
        s3 = boto3.client('s3')
        res = s3.list_objects_v2(
            Bucket=bucket_name,
            Prefix=path_to_folder
        )
        return 'Contents' in res
    except ClientError as e:
        # Logic to handle errors.
        raise e

As of now, the list_objects_v2's response dictionary will not have a 'Contents' key if the prefix is not found (docs).

feran
  • 293
  • 3
  • 9