AWS S3 presigned POST returns different responses

Question

I have an EC2 instance and an AMI for this instance deployed (I will call it Original and Clone to differentiate them somehow). Theoretically, it should work the same way but I found a difference between presigned responses.

So the response on Original is:

{
    "url": "https://BUCKET1.s3.amazonaws.com/",
    "fields": {
        "AWSAccessKeyId": "***",
        "key": "images/586d43bde4d2a1d910bc63d0b15691a0.png",
        "policy": "***",
        "signature": "***"
    }
}

And on Clone I see the following response:

{
    "url": "https://BUCKET2.s3.amazonaws.com/", 
    "fields": {
        "key": "images/e980a22ef60d3ac7b41324cfa354b45b.jpg", 
        "x-amz-algorithm": "AWS4-HMAC-SHA256", 
        "x-amz-credential": "***/20220801/us-east-2/s3/aws4_request", 
        "x-amz-date": "20220801T133442Z",
        "policy": "***", 
        "x-amz-signature": "***"
    }
}

I think there is a difference between BUCKET1 and BUCKET2 settings and I tried to use BUCKET2 on Original and expected to receive the response with X-prefixed keys, but received the response without them. So I don't have an idea what's wrong...

UPD: both instances have boto3==1.15.12 installed

UPD: logs:

Original:

2022-08-02 13:20:24,618 botocore.hooks [DEBUG] Event choose-signer.s3.PutObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f0ae8e4bf10>>
2022-08-02 13:20:24,618 botocore.hooks [DEBUG] Event choose-signer.s3.PutObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f0ae8e4bf10>>
2022-08-02T13:20:24 - [DEBUG] - Event choose-signer.s3.PutObject: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f0ae8e4bf10>>
2022-08-02T13:20:24 - [DEBUG] - Event before-sign.s3.PutObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f0ae8a69b50>>
2022-08-02T13:20:24 - [DEBUG] - Checking for DNS compatible bucket for: https://s3.amazonaws.com/BUCKET1
2022-08-02 13:20:24,618 botocore.hooks [DEBUG] Event before-sign.s3.PutObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f0ae8a69b50>>
2022-08-02 13:20:24,618 botocore.hooks [DEBUG] Event before-sign.s3.PutObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f0ae8a69b50>>
2022-08-02 13:20:24,618 botocore.utils [DEBUG] Checking for DNS compatible bucket for: https://s3.amazonaws.com/BUCKET1
2022-08-02 13:20:24,618 botocore.utils [DEBUG] Checking for DNS compatible bucket for: https://s3.amazonaws.com/BUCKET1
2022-08-02T13:20:24 - [DEBUG] - URI updated to: https://BUCKET1.s3.amazonaws.com/
2022-08-02 13:20:24,618 botocore.utils [DEBUG] URI updated to: https://BUCKET1.s3.amazonaws.com/
2022-08-02 13:20:24,618 botocore.utils [DEBUG] URI updated to: https://BUCKET1.s3.amazonaws.com/

Clone:

2022-08-02T13:25:11 - [DEBUG] - Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2022-08-02 13:25:11,154 botocore.hooks [DEBUG] Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2022-08-02T13:25:11 - [DEBUG] - Changing event name from before-call.apigateway to before-call.api-gateway
2022-08-02 13:25:11,156 botocore.hooks [DEBUG] Changing event name from before-call.apigateway to before-call.api-gateway
2022-08-02T13:25:11 - [DEBUG] - Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2022-08-02 13:25:11,157 botocore.hooks [DEBUG] Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2022-08-02T13:25:11 - [DEBUG] - Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2022-08-02 13:25:11,159 botocore.hooks [DEBUG] Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2022-08-02 13:25:11,159 botocore.hooks [DEBUG] Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2022-08-02T13:25:11 - [DEBUG] - Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2022-08-02T13:25:11 - [DEBUG] - Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2022-08-02 13:25:11,160 botocore.hooks [DEBUG] Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2022-08-02T13:25:11 - [DEBUG] - Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2022-08-02 13:25:11,162 botocore.hooks [DEBUG] Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2022-08-02T13:25:11 - [DEBUG] - Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2022-08-02 13:25:11,165 botocore.hooks [DEBUG] Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
2022-08-02 13:25:11,165 botocore.hooks [DEBUG] Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2022-08-02T13:25:11 - [DEBUG] - Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
2022-08-02 13:25:11,165 botocore.hooks [DEBUG] Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2022-08-02T13:25:11 - [DEBUG] - Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2022-08-02T13:25:11 - [DEBUG] - Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2022-08-02 13:25:11,166 botocore.hooks [DEBUG] Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2022-08-02T13:25:11 - [DEBUG] - Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/endpoints.json
2022-08-02 13:25:11,168 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/endpoints.json
2022-08-02T13:25:11 - [DEBUG] - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7eff768ee8c0>
2022-08-02 13:25:11,174 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7eff768ee8c0>
2022-08-02T13:25:11 - [DEBUG] - Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/s3/2006-03-01/service-2.json
2022-08-02 13:25:11,184 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/s3/2006-03-01/service-2.json
2022-08-02 13:25:11,194 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7eff7691a440>
2022-08-02T13:25:11 - [DEBUG] - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x7eff7691a440>
2022-08-02T13:25:11 - [DEBUG] - Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7eff749d2950>
2022-08-02 13:25:11,194 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function lazy_call.<locals>._handler at 0x7eff749d2950>
2022-08-02T13:25:11 - [DEBUG] - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7eff7691a200>
2022-08-02 13:25:11,203 botocore.hooks [DEBUG] Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x7eff7691a200>
2022-08-02T13:25:11 - [DEBUG] - Setting s3 timeout as (60, 60)
2022-08-02 13:25:11,205 botocore.endpoint [DEBUG] Setting s3 timeout as (60, 60)
2022-08-02 13:25:11,207 botocore.loaders [DEBUG] Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/_retry.json
2022-08-02T13:25:11 - [DEBUG] - Loading JSON file: /usr/local/lib/python3.7/site-packages/botocore/data/_retry.json
2022-08-02T13:25:11 - [DEBUG] - Registering retry handlers for service: s3
2022-08-02 13:25:11,207 botocore.client [DEBUG] Registering retry handlers for service: s3
2022-08-02T13:25:11 - [DEBUG] - Event choose-signer.s3.PutObject: calling handler <function set_operation_specific_signer at 0x7eff7688c9e0>
2022-08-02 13:25:11,209 botocore.hooks [DEBUG] Event choose-signer.s3.PutObject: calling handler <function set_operation_specific_signer at 0x7eff7688c9e0>
2022-08-02T13:25:11 - [DEBUG] - Event before-sign.s3.PutObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7eff6fd51ed0>>
2022-08-02T13:25:11 - [DEBUG] - Defaulting to S3 virtual host style addressing with path style addressing fallback.
2022-08-02 13:25:11,209 botocore.hooks [DEBUG] Event before-sign.s3.PutObject: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7eff6fd51ed0>>
2022-08-02 13:25:11,210 botocore.utils [DEBUG] Defaulting to S3 virtual host style addressing with path style addressing fallback.
2022-08-02T13:25:11 - [DEBUG] - Checking for DNS compatible bucket for: https://s3.us-east-2.amazonaws.com/BUCKET2
2022-08-02T13:25:11 - [DEBUG] - URI updated to: https://BUCKET2.s3.amazonaws.com/
2022-08-02 13:25:11,210 botocore.utils [DEBUG] Checking for DNS compatible bucket for: https://s3.us-east-2.amazonaws.com/BUCKET2
2022-08-02 13:25:11,210 botocore.utils [DEBUG] URI updated to: https://BUCKET2.s3.amazonaws.com/

jarmod · Accepted Answer · 2022-08-02T13:58:13.430

1

I think the difference is probably that boto3's default signing method is sigv2 for older regions and sigv4 for all newer regions (those that came online after sigv4 became the default). It looks like bucket1 may be in us-east-1 (older region) while bucket2 is in us-east-2 (newer region).

Enforce sigv4 explicitly in your code as follows:

# Get the service client with sigv4 configured
s3 = boto3.client('s3', config=Config(signature_version='s3v4'))

Also, you're using boto3 1.15.12 which is almost 2 years old. You should update this.

edited Aug 02 '22 at 13:58

answered Aug 02 '22 at 13:47

jarmod

71,565
16
115
122

yes, my buckets are in us-east-1 & us-east-2 and I didn't know this nuance with signature version. also updated boto3. thanks! – hustas88 Aug 02 '22 at 16:26

AWS S3 presigned POST returns different responses

1 Answers1