How do I upload a file into an Amazon S3 signed url with python?

Question

I have managed to generate an Amazon S3 Signed URL with boto3.

import logging
import boto3
from botocore.exceptions import ClientError

def create_presigned_url(bucket_name, object_name, expiration=3600):
    """Generate a presigned URL to share an S3 object

    :param bucket_name: string
    :param object_name: string
    :param expiration: Time in seconds for the presigned URL to remain valid - 1 hour in this case
    :return: Presigned URL as string. If error, returns None.
    """

    # Generate a presigned URL for the S3 object
    s3_client = boto3.client('s3')
    try:
        response = s3_client.generate_presigned_url('get_object', Params={'Bucket': bucket_name, 'Key': object_name}, ExpiresIn=expiration)
    except ClientError as e:
        logging.error(e)
        return None

    # The response contains the presigned URL
    return response

result = create_presigned_url("bucket123", "test.txt")

assuming bucket123 exists. This code returns me a URL like this:

https://bucket123.s3.amazonaws.com/test.txt?AWSAccessKeyId=AKIA56EZ34KPFKO5366J&Signature=7aAZXXDwtvKGLaeu1ATHRJuaFCs%3D&Expires=1635117707

My question is:

Using python and boto3, is it possible to upload a file in this pre-signed url? How do I go on about this?

If you have boto3 then you would not use a pre-signed URL to upload a file to S3. You would simply use the boto3 methods, such as put_object and upload_file. — jarmod, Oct 25 '21 at 02:03
@jarmod While I agree with you, ... I need to be able to provide a pre-signed URL in this case. How would you suggest I get a URL that looks like this back? `https://bekim-cribl1.s3.amazonaws.com/test.tgz?AWSAccessKeyId=AKIA56EZ34KPFKO5367J&Signature=MUUJWwA%2B9V5d7H7DjeorLYsSguo%3d&Expires=1635111412` — David, Oct 26 '21 at 06:14
@Marcin Thank you so much for the help with this .. Ideally I would like to have the URL generated back in the fashion of : `https://bekim-cribl1.s3.amazonaws.com/test.tgz?AWSAccessKeyId=AKIA56EZ34KPFKO5367J&Signature=MUUJWwA%2B9V5d7H7DjeorLYsSguo%3d&Expires=1635111412` Any idea on how can I achieve that? — David, Oct 26 '21 at 06:15
Why do you want the pre-signed URL to look a certain way (which appears to be sigv2)? The currently-supported signing method is signature v4. Earlier signing methods are in an extended period of deprecation. — jarmod, Oct 26 '21 at 09:34
@jarmod You make a good point. Pardon me for asking just like this, but, How do I obtain a Signature v4 type of URL? — David, Oct 26 '21 at 15:28
All current SDKs and the awscli will generate v4 signed URLs, by default. — jarmod, Oct 26 '21 at 15:37

Marcin · Accepted Answer · 2021-10-24T23:53:58.393

Once you have the pre-signed url, you don't need boto3 at all. Instead you can use regular python functions for uploading files to S3. For example using python's requests. Also you should be using generate_presigned_post


def create_presigned_post(bucket_name, object_name,
                          fields=None, conditions=None, expiration=3600):
    """Generate a presigned URL S3 POST request to upload a file

    :param bucket_name: string
    :param object_name: string
    :param fields: Dictionary of prefilled form fields
    :param conditions: List of conditions to include in the policy
    :param expiration: Time in seconds for the presigned URL to remain valid
    :return: Dictionary with the following keys:
        url: URL to post to
        fields: Dictionary of form fields and values to submit with the POST
    :return: None if error.
    """

    # Generate a presigned S3 POST URL
    s3_client = boto3.client('s3')
    try:
        response = s3_client.generate_presigned_post(bucket_name,
                                                     object_name,
                                                     Fields=fields,
                                                     Conditions=conditions,
                                                     ExpiresIn=expiration)
    except ClientError as e:
        logging.error(e)
        return None

    # The response contains the presigned URL and required fields
    return response

result = create_presigned_post("bucket123", "test.txt")



import requests    # To install: pip install requests

# Generate a presigned S3 POST URL
object_name = 'test.txt'
result = create_presigned_post("marcin3", object_name)

# Demonstrate how another Python program can use the presigned URL to upload a file
with open(object_name, 'rb') as f:
    files = {'file': (object_name, f)}
    http_response = requests.post(result['url'], data=result['fields'], files=files)
# If successful, returns HTTP status code 204
logging.info(f'File upload HTTP status code: {http_response.status_code}')

Gettting error: ` SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your key and signing method.AKIA56EZ34KPFKO53663POST multipart/form-data; boundary=f9cd89cc9f8848cb3c648bb60896faae 1635120065 /bucket123/test.txttCJ1oWjPRcRs3vaP64KzxFgn1z4= ` @Marcin — David, Oct 24 '21 at 23:04
error continued: `50 4f 53 54 0a 0a 6d 75 6c 74 69 70 61 72 74 2f 66 6f 72 6d 2d 64 61 74 61 3b 20 62 6f 75 6e 64 61 72 79 3d 66 39 63 64 38 39 63 63 39 66 38 38 34 38 63 62 33 63 36 34 38 62 62 36 30 38 39 36 66 61 61 65 0a 31 36 33 35 31 32 30 30 36 35 0a 2f 62 65 6b 69 6d 2d 63 72 69 62 6c 31 2f 74 65 73 74 2e 74 78 743D8J1523P6FS7TBVtnERzMP+wJsbPQcSpxiVhhdWuXFSmWkZDK+qkBKycd3ilLFHJa5FrrwBuwhfe7lz09e3vgX+Tkm=` — David, Oct 24 '21 at 23:04
The file finally uploads fine with the create_presigned_post() method. Two final remarks. I don't see the pre-signed URL printed anymore AND When I go to access the file in the S3 console, I get ` AccessDenied Access Denied CQNCGDS4YKCPDWQ6 LsJI1jBMUCeQpL4qjfqipmvYeCJ5TaS3ZoBwbmXMbbklWo5stWMdVtGI50Ib+XDciHerIm5AADw= ` — David, Oct 26 '21 at 06:11
@Macin : I asked the access denied part here: https://stackoverflow.com/q/69718314/779429 — David, Oct 26 '21 at 06:26

How do I upload a file into an Amazon S3 signed url with python?

1 Answers1