Write to a specific folder in an S3 bucket using boto3 python

Question

The question

I have a file that I would like to write to a specific folder within my S3 bucket lets call the bucket bucket and the folder folderInBucket

I am using the boto3 library to achieve and have the following function:

What I have done

def upload_file(file_name, bucket, object_name = None):
    
    if object_name is None:
        object_name = file_name
    s3 = b3.client('s3')
    try:
        response = s3.upload_file(file_name, bucket, Key='bulk_movies.json')
    except ClientError as e:
        print(e)
        return False
    print('Success!')
    return True

upload_file('./s3/bulk_movies.json', 'bucket')

I have also tried when calling the function using bucket/folderInBucket as the second parameter but this produces an error in the code (sort of as expected actually)

Gaps in understanding

This function was more or less ripped from the boto3 documentation. The docs don't really specify how to write into a specific folder within our S3 bucket. I know for sure the file itself is able to write fine into the bucket's main directory because the code outlined above works without issue.

This is not a direct answer to your question, but I would recommend checking out `fsspec`: https://filesystem-spec.readthedocs.io/en/latest/?badge=latest. See here for example: https://filesystem-spec.readthedocs.io/en/latest/features.html?highlight=s3%3A%2F%2F#remote-write-caching — anon01, Sep 10 '21 at 19:45
you are passing `Key` as just `bulk_movies.json` - there is your problem. I suggest adding `Key` as a configurable param to the function. Try what happens if you pass in `Key='path/to/bulk_movies.json'` — rv.kvetch, Sep 10 '21 at 20:01
also, probably related, you're assigning `object_name` but not using it anywhere — rv.kvetch, Sep 10 '21 at 20:02

score 2 · Accepted Answer · answered Sep 10 '21 at 20:07

I was able to get this to work by modifying your function and the call a bit to include the object_name as the absolute path to the file on the bucket.

import boto3 as b3

def upload_file(file_name, bucket, object_name):
    if object_name is None:
        object_name = file_name
    s3 = b3.client('s3')
    try:
        response = s3.upload_file(file_name, bucket, Key=object_name)
    except ClientError as e:
        print(e)
        return False
    print('Success!')
    return True

upload_file('bulk_movies.json', '<bucket-name>', 'folderInBucket/bulk_movies.json')

Please share the error if you're still running into one. As far as the file upload is concerned, it should work with what you have already done.

score 0 · Answer 2 · answered Sep 10 '21 at 20:00

S3 is "blob storage" and as such the concept of a "folder" doesn't really exist. When uploading a file you just provide the complete prefix to the destination string.

Here's an example:

import boto3

s3 = boto3.resource('s3')
folder_name = 'folder_name'
s3.meta.client.upload_file('helloworld.txt', 'bucketname', folder_name + '/helloworld.txt')

score 0 · Answer 3 · answered Sep 10 '21 at 20:03

0

There is no such thing - "folder" is s3.
What do you consider as "folder" is part of the file name.

URL example: s3://mybucket/folder1/folder2/file.txt

In the example above the file name is 'folder1/folder2/file.txt'

answered Sep 10 '21 at 20:03

balderman

22,927
7
34
52

1

More common terminology is that `folder1/folder2/file.txt` is an object key (rather than filename), and `folder1/folder2/` is an object key prefix. – jarmod Sep 10 '21 at 20:06

score 0 · Answer 4 · answered Sep 16 '21 at 15:16

buckets are not folders, but they act like folders. you store objects in a s3 bucket. An object has a key. The file is uploaded by default with private permissions. you can set the acl to public-read allowing everyone in the world to see the file.

s3.upload_file(
    Bucket='gid-requests',
    Filename='bulk_movies.json',
    Key='folderInBucket/bulk_movies.json',
    ExtraArgs={'ACL':'public-read'})

Write to a specific folder in an S3 bucket using boto3 python

4 Answers4