1

new coder here. For work, I receive a request to put certain files in an already established s3 bucket with a requested "path."

For example: "Create a path of (bucket name)/1/2/3/ with folder3 containing (requested files)"

I'm looking to create a Python3 script to upload multiple files from my local machine to a specified bucket and "path" using CLI arguments specifying the file(s), bucket name, and "path"/key - I understand s3 doesn't technically have a folder structure, and that you have to put your "folders" in as part of the key, which is why I put "path" in quotes.

I have a working script doing what I want it to do, but the bucket/key is hard coded at the moment and I'm looking to get away from that with the use and understanding of CLI arguments. This is what I have so far -- it just doesn't upload the file, though it builds the path in s3 successfully :/

EDIT: Below is the working version of what I was looking for!

import argparse
#import os
import boto3


def upload_to_s3(file_name, bucket, path):
    s3 = boto3.client('s3')
    s3.upload_file(file_name, bucket, path)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument('--file_name')
    parser.add_argument('--bucket')
    parser.add_argument('--path')
    args = parser.parse_args()

    upload_to_s3(args.file_name, args.bucket, args.path)

my input is:

>>> python3 s3_upload_args_experiment.py --file_name test.txt --bucket mybucket2112 --path 1/2/3/test.txt

Everything executes properly!

Thank you much!

user12765410
  • 41
  • 1
  • 5
  • What is your actual question? FYI, you should use `s3_path` in `put_object()` so that it includes the filename. The `Key` is the path _plus_ filename, not just the path. Is there any reason you don't just use the [AWS Command-Line Interface (CLI)](http://aws.amazon.com/cli/) to do the upload? – John Rotenstein Jun 09 '20 at 21:00
  • Thanks for your response. Sorry, should have clarified that my main purpose for this was more for my own learning in how to do this in a script format, though totally viable otherwise using the CLI. In short, I'm trying to create a version of the the answer in this article but in its most basic form, ie without renaming and reformatting, or incorporating `datetime` https://stackoverflow.com/questions/61291816/how-to-implement-argparse-in-python I suppose I'm just looking for a Python version of what the AWS CLI already accomplishes, but using custom arguments – user12765410 Jun 09 '20 at 22:07
  • BTW, no need to create the folders. Just upload the file to the desired path (in the Key) and it will magically work! – John Rotenstein Jun 10 '20 at 03:59
  • ah! yes, right - thank you. do you have any suggestions for how I may be able to implement that with argparser, taking the `filename` , `bucketname`, and `path` using either positional or optional arguments? @JohnRotenstein – user12765410 Jun 10 '20 at 04:35
  • Update: everything seems to work except the actual file upload! Please see edited code above – user12765410 Jun 10 '20 at 05:16
  • Works fine for me, but remember that the `--path` needs to include the name of the object too (eg `--path 1/2/3/foo.txt`). The Key is not the destination directory, it is the **full name** of the object, _including_ the path. – John Rotenstein Jun 10 '20 at 05:41
  • @JohnRotenstein Thank you so much for the clarification! Truly appreciate you. – user12765410 Jun 10 '20 at 05:49

0 Answers0