4

There have been a few other questions like this, but none seem to have answered my problem specifically. I have a folder full of files I want to upload to Amazon S3, and I have tried various different things, all leading to no upload. Here's the code I have now. It successfully creates the folder on S3, but doesn't upload.

import os
import boto3

s3_bucket = 'bucketname'
s3_bucket_region = 'us-east-1'
folder = 'FolderName'

key_name = folder + '/' 
s3_connect = boto3.client('s3', s3_bucket_region)
try:
    bucket = s3_connect.put_object(Bucket=s3_bucket, Key=key_name)
    print "Bucket:", bucket
except Exception as e:
    print "Bucket Error " , e

# upload File to S3
for filename in os.listdir(folder):

    file_key_name = folder + '/' + filename
    local_path = os.getcwd()
    local_name = local_path + '/' + key_name + filename
    upload = s3_connect.upload_file(local_name, s3_bucket, file_key_name)

I have tried a file_key_name of just something like 'FolderName/' as well as 'FolderName/FileName'. I thought at first the first argument (local_name) didn't need a full path, but a relative path would work, but neither a full or relative path works.

What am I missing? The docs are pretty woefully lacking in this department. Plus it doesn't throw an error, it just doesn't upload.

maxpearl
  • 75
  • 1
  • 8
  • This might be useful https://techietweak.wordpress.com/2016/05/16/file-handling-in-aws-s3-with-python-boto-library/ in this line `k.key=fileName` you should put `k.key=FolderName/FileName` for it to be uploaded to the folder. – moondaisy Apr 13 '17 at 21:05
  • I came across that, but alas, that uses the boto library, and I'm using boto3. – maxpearl Apr 14 '17 at 00:49
  • My bad, have you tried with the 3 examples in the doc? They all call `boto3.resource` instead of `boto3.client`. This is the bucket one: http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Bucket.upload_file This is the object one: http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Object.upload_file Using client (not sure if the same client as yours): http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.upload_file – moondaisy Apr 14 '17 at 00:59
  • Have you considered simply using the [AWS Command-Line Interface (CLI)](http://aws.amazon.com/cli/)? There is a `aws s3 sync` command that can copy a hierarchy of directories with one command. – John Rotenstein Apr 14 '17 at 03:28

1 Answers1

3

Your code works perfectly fine for me, but you need to remove the try/catch section that creates the folder on Amazon S3.

Amazon S3 does not actually support folders. Rather, the Key of an object includes the full path (eg images/foo.jpg). Therefore, you don't need to create the folders before storing a file in them (since they do not exist!).

Your code will have difficulty with nested folders. You might consider using the aws s3 sync or aws s3 cp commands in the AWS Command-Line Interface (CLI) if you don't need it written in your own code.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • That was it. Thanks! Luckily I don't need nested folders, just one. And it's part of a larger application, so it has to be in python. – maxpearl Apr 17 '17 at 17:43