Can't upload a file to S3 with pre-signed URL no matter what I do. AWS command line works. CURL and anything else = 403

Question

My AWS credentials in ~/.aws/credentials are correct and working. Proof?

$ aws s3api put-object --bucket <my bucket name> --key videos/uploads/yoda.jpeg --body /Users/r<my_name>/Desktop/Archive/yoda.jpeg

getting back:

{
    "ETag": "\"66bee0b7caf3d127900e0a70f2da4b5f\""
}

The upload worked from command line. And I can see my file when I see my S3 bucket in AWS's management console.

NOW- I delete the successfully uploaded file from S3 and I'm trying to upload it again, this time via a presigned URL

$ aws s3 presign s3://<my-bucket>/videos/uploads/yoda.jpeg

for which I get:

https://<my-bucket>.s3.amazonaws.com/videos/uploads/yoda.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=<MY-AWS-KEY-ID>%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20210207T222859Z&X-Amz-Signature=3a3624b9e264c119ebdf93c989efb73337f7ab8793e89554c7b000e1fc93c85c

From this moment on, any PUT attempt with CURL, POSTMAN or any other tool, with this URL fails to upload the file always ends up with 403 (yes, it's not expiring, it fails immediately) and The request signature we calculated does not match the signature you provided is the excuse provided by AWS.

The S3 bucket has a policy allowing the user whose credentials are in /.aws/credentials to Put* on that very bucket.

What is going on? Why doesn't pre-signed URL work?

CURL ATTEMPT

$ curl --location --request PUT 'https://<my-bucket-name>.s3.amazonaws.com/videos/uploads/yoda.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=<MY-AWS-KEY-ID>%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20210207T224403Z&X-Amz-Signature=8a8625591e6c4e0871f97bf5e15c2f93b3e373cfc1c2daddb2cf34edb10a5670%0A' \
--header 'Content-Type: image/jpeg' \
--data-binary '@/Users/<MY-NAME>/Desktop/Archive/yoda.jpeg'

to which I get:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
    <AWSAccessKeyId><---MY--ACCESS--KEY--ID--->/AWSAccessKeyId>
    <StringToSign>AWS4-HMAC-SHA256
20210207T224403Z
20210207/us-east-2/s3/aws4_request
da93cc1a0ec196fe0726ec6d5cace8c1b2b4865b20663bf0240454e276dbef6f</StringToSign>
    <SignatureProvided>8a8625591e6c4e0871f97bf5e15c2f93b3e373cfc1c2daddb2cf34edb10a5670
</SignatureProvided>
    <StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 31 30 32 30 37 54 32 32 34 34 30 33 5a 0a 32 30 32 31 30 32 30 37 2f 75 73 2d 65 61 73 74 2d 32 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 64 61 39 33 63 63 31 61 30 65 63 31 39 36 66 65 30 37 32 36 65 63 36 64 35 63 61 63 65 38 63 31 62 32 62 34 38 36 35 62 32 30 36 36 33 62 66 30 32 34 30 34 35 34 65 32 37 36 64 62 65 66 36 66</StringToSignBytes>
    <CanonicalRequest>PUT
/videos/uploads/yoda.jpeg
X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=<---MY--ACCESS--KEY--ID--->%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&amp;X-Amz-Date=20210207T224403Z&amp;X-Amz-Expires=3600&amp;X-Amz-SignedHeaders=host
host:<my-bucket-name>.s3.amazonaws.com

host
UNSIGNED-PAYLOAD</CanonicalRequest>
    <CanonicalRequestBytes>50 55 54 0a 2f 76 69 64 65 6f 73 2f 75 70 6c 6f 61 64 73 2f 79 6f 64 61 2e 6a 70 65 67 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 51 33 44 34 36 52 4e 50 48 51 4e 4b 47 42 46 4b 25 32 46 32 30 32 31 30 32 30 37 25 32 46 75 73 2d 65 61 73 74 2d 32 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 31 30 32 30 37 54 32 32 34 34 30 33 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 36 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 0a 68 6f 73 74 3a 6c 73 74 76 32 2d 70 75 62 6c 69 63 2e 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes>
    <RequestId>CBJT0Y4SX9A7RB26</RequestId>
    <HostId>h+5b/u8cdi34yuSDBX0Z/mZGQMtRZIMS4rvIwiKzOZSOZhRoQfak8cOdVBq2BgtU1qbqlHrO2TY=</HostId>
</Error>

TRYING TO GENERATE THE PRESIGN URL FROM PYTHON. STILL DOES NOT WORK. THE URL IS FAULTY- AWS REJECTS WITH THE SAME 403

 def get_upload_pre_signed_url(bucket_name, object_name, expiration=3600):
        s3_client = boto3.client('s3')
        try:
            response = s3_client.generate_presigned_url('put_object',
                                                        Params={'Bucket': bucket_name,
                                                                'Key': object_name},
                                                        ExpiresIn=expiration)
        except ClientError as e:
            return None
    
            # The response contains the presigned URL
        return response

URL generated from this:

https://<my-bucket>.s3.amazonaws.com//videos/uploads/yoda.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<my-AWS-KEY-ID>%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210207T231306Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=968a3e2cab9b7e907df69e24aae24d79ea40f52a52d407591d7cbd69c86fe67b

Curling it gets the same 403. Doesn't change.

Can you show the entire curl command and full output? – Robert Hafner Feb 07 '21 at 22:41 — Robert Hafner, Feb 07 '21 at 22:41

Robert Hafner · Answer 1 · 2021-02-07T23:17:51.597

2

The aws s3 presign command creates URLs that can be used for downloading files. It does not create URLs that can be used for uploading. To quote the docs-

Generate a pre-signed URL for an Amazon S3 object. This allows anyone who receives the pre-signed URL to retrieve the S3 object with an HTTP GET request. All presigned URL’s now use sigv4 so the region needs to be configured explicitly.

To create upload URLs you need to jump out of the command line and into your language of choice and use the full AWS SDK.

Python example:

s3.generate_presigned_post(
        Bucket=BUCKET_NAME,
        Key=FILE_KEY,
        ExpiresIn=(5*60)
    )

Note that it uses the generate_presigned_post function to do this.

edited Feb 07 '21 at 23:17

answered Feb 07 '21 at 22:49

Robert Hafner

3,364
18
23

this still gives 403, but now with a simple "Access Denied" -- no more "signature doesn't match" – JasonGenX Feb 07 '21 at 23:33
Does your config include a default region? If not try adding it. On the curl side make sure your command looks like `curl -v --upload-file ${fileName} ${location}` so the appropriate upload headers are added. – Robert Hafner Feb 07 '21 at 23:37
still 403 access denied, but this time without the "signature doesn't match" – JasonGenX Feb 07 '21 at 23:38
if the target bucket has KMS encryption you need to include the header 'x-amz-server-side-encryption' = 'aws:kms' to the POST request – Matt Cliff Oct 28 '21 at 22:29

score 2 · Accepted Answer · answered Feb 09 '21 at 20:40

SOLUTION: Turns out the Boto3 I was using wasn't up to date and I was using it wrong. After fixing those, the code that worked for me was:

# THE CREDENTIALS ARE PART OF MY TESTING CODE. NO WORRIES THEY'RE IN AN ENV VARIABLE NOW

def get_upload_pre_signed_url(bucket_name, key, expiration=3600):
    s3 = boto3.client('s3',
                      aws_access_key_id="<my access_key_id",
                      aws_secret_access_key="<my_secreet_access_key>",
                      config=Config(region_name='us-east-2', s3.{"use_accelerate_endpoint": True}))

    try:
        url = s3.generate_presigned_url('put_object', Params={'Bucket': bucket_name, 'Key': key},
                                        ExpiresIn=expiration,
                                        HttpMethod='PUT')
    except ClientError as e:
        return None

    return url

Balu Vyamajala · Answer 3 · 2021-02-07T23:46:14.830

1

s3 presign is only for generating url for download

Generate a pre-signed URL for an Amazon S3 object. This allows anyone who receives the pre-signed URL to retrieve the S3 object with an HTTP GET request. All presigned URL’s now use sigv4 so the region needs to be configured explicitly

we will still need to specify Bucket and Key to which we want to upload.

NodeJs:

const bucketParms = {
  Bucket: "sample-temp-bucket",
  Key: "HeadShot.jpg",
  ContentType:'image/jpeg'
};

s3.getSignedUrl("putObject", bucketParms, (error, url) => {
  if (error) console.log("error", error);
  if (url) console.log("url", url);
});

Python:

response = s3_client.generate_presigned_url('put_object',
                                                Params={'Bucket': bucket_name,
                                                        'Key': object_name,
                                                        'ContentType':'image/jpeg'},
                                                ExpiresIn=expiration)

we can do a curl or postman

curl --location --request PUT 'https://test-events.s3.amazonaws.com/98..?X....' \
--header 'Content-Type: image/jpeg' \
--data-binary '/Users/user/image/path'

edited Feb 07 '21 at 23:46

answered Feb 07 '21 at 22:54

Balu Vyamajala

9,287
1
20
42

Does't help. I am now generating my url from python and it still gives out a 403. See code in the message above – JasonGenX Feb 07 '21 at 23:11
@JasonGenX - i updated my answer with a python example. – Robert Hafner Feb 07 '21 at 23:18
@JasonGenX Please generate the url including the content type header. – Balu Vyamajala Feb 07 '21 at 23:26
doesn't work. same response. Didn't you suggest to use generate_presigned_post? you'v changed to generate_presigned_url? – JasonGenX Feb 07 '21 at 23:31
No. I had NodeJs example, changed to Python(probably the other answer). I just tried this, works for me. when trying from postman/curl the key has to match with file name. can you confirm that you have it same. – Balu Vyamajala Feb 07 '21 at 23:37

score 1 · Answer 4 · answered Feb 07 '21 at 22:56

The presign cli command is only for GET requests. If you need anything else, you have to use AWS API directly - as suggested below: you can use a short python script for that. We used for one of our applications a lambda which you can call and you will get the right url. Also, the presigned URL uses the role which called the API, so it has the same permissions. Including the fact that if you are using and STS assumed role and the grant expires sooner than the expiration time of the presigned url, the url will still fail. But if you use regular roles (like your aws cli profile), it should be ok.

https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3/presign.html

Generate a pre-signed URL for an Amazon S3 object. This allows anyone who receives the pre-signed URL to retrieve the S3 object with an HTTP GET request. All presigned URL’s now use sigv4 so the region needs to be configured explicitly.

Possible to send a PUT request to aws s3 presign url?

AWS CLI doesn't support presigned PUT URL yet. You can easily generate one using Python Boto3 though. The documentation is here. If you want a presigned PUT, you just need to let ClientMethod param be put_object.

score 1 · Answer 5 · answered Feb 08 '21 at 03:14

Your edit is correct for generating the URL using the SDK.

That said, to use the URL, the HTTP headers curl uses need to be exactly correct. Notably here, the signature requires there be no Content-Type header sent to the server. Since --data-binary forces one, the easiest way I know of the get curl to do the right thing is use the --upload-file flag:

$ curl $URL --upload-file yoda.jpg

score 0 · Answer 6 · answered Feb 09 '21 at 08:09

It was not working for me too, it was returning 403 forbidden. Things that helped were:

Setting the signature to v4 new AWS.S3({ signatureVersion: "v4" })
If you use Metadata, do encode the content encodeURI("meta text with symbols#@");

here is more coupled ts/js code:

const s3 = new AWS.S3({ signatureVersion: "v4" });
const s3Params = {
    Bucket: <BUCKET_NAME>,
    Key: <KEY>,
    Expires: 300,
    ContentType: <CONTENT_TYPE>,
    Metadata: {
        title: encodeURI('SomeTitle with utf-8 characters'),
    },
};
const signedPutUrl = await s3.getSignedUrlPromise(
    "putObject",
    s3Params
);
console.log('the signed url',signedPutUrl);
return {'url':signedPutUrl};

Can't upload a file to S3 with pre-signed URL no matter what I do. AWS command line works. CURL and anything else = 403

6 Answers6

Linked