2

I keep on failing to use signed URLs for S3 objects residing in a bucket in the ap-east-1 (Hong Kong) AWS region.

Specifically, I first set the signature version to V4 (see https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version) like so:

aws configure set default.s3.signature_version s3v4

And then create a signed URL with this command:

aws s3 presign --region=ap-east-1 s3://<name of bucket in ap-east-1 region>/<object name>

When I test the resulting URL:

curl -i "https://<bucket name>.s3.amazonaws.com/<object name>?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...%2Fap-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190928T034534Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=..."

...I get an HTTP/400 response with the body set to:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
   <Code>IllegalLocationConstraintException</Code>
   <Message>The ap-east-1 location constraint is incompatible for the region specific endpoint this request was sent to. 
   </Message>
   <RequestId>...</RequestId>
<HostId>

Interestingly, when doing the same for an object in another bucket that resides in the us-west-2 (Oregon) AWS region, the resulting signed URL works fine.

Has anyone run into this problem? I don't quite know how to parse the error message, but I do wonder if I need to set the --endpoint-url configuration setting and if so, to what?

Ideas anyone?

Thanks, Soeren

Soeren
  • 691
  • 5
  • 12
  • Hong Kong was the first region where some new S3 behavior rolled out, and S3 doesn't work exactly the same there as it does in other regions.. but the SDK *should* understand the rules, so my first question os whether you are using the latest version of the SDK. – Michael - sqlbot Sep 28 '19 at 16:03
  • Thanks, Michael - I believe I am. This is what I'm on: ``` aws-cli/1.16.248 Python/3.6.8 Linux/4.19.67-microsoft-standard botocore/1.12.238 ``` This should currently be the latest version. I am aware of the fact that they are using V4 signatures in this region. But that's about the only difference that I am aware of. – Soeren Sep 28 '19 at 23:35
  • 2
    You're right about Sig V4, but there's another change: [*"Buckets created in Regions launched after March 20, 2019 are not reachable via the `https://bucket.s3.amazonaws.com` naming scheme."*](https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html) What they don't mention is that those URLs are still present for new region buckets, but unusable, since all they do is return the confusing `IllegalLocationConstraintException` error. I suspect they finally realized that the generic endpoints were pointless since signing Sig V4 requires knowledge of the region anyway. (Sig V2 did not.) – Michael - sqlbot Sep 29 '19 at 18:42

1 Answers1

2

You need to set the addressing style in your CLI configuration for this to work. See AWS CLI S3 Configuration documentation for details.

Set of steps:

aws configure set default.s3.addressing_style virtual
aws s3 presign s3://<bucket-in-ap-east-1>/<object_key> --region ap-east-1

Equivalent code in Python

import boto3
def generate_presigned_url(bucket_region, bucket_name, object_key, expiration):
   s3_client = boto3.client('s3', region_name=bucket_region,  config=boto3.session.Config(s3={'addressing_style': 'virtual'}, signature_version='s3v4'))
   response = s3_client.generate_presigned_url('get_object', Params={'Bucket': bucket_name, 'Key': object_key}, ExpiresIn=expiration)
   return response

Usage

generate_presigned_url('ap-east-1', bucket_name, object_name, expiration)

Sample output

https://your-bucket.s3.ap-east-1.amazonaws.com/object-key?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXXX%2Fap-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200714T050137Z&X-Amz-Expires=7200&X-Amz-SignedHeaders=host&X-Amz-Security-Token=TTTT&X-Amz-Signature=cde17576df4502854639bdd52fdd493e3b4b946fd668b8478c70be0d294df78d'

vsnyc
  • 2,117
  • 22
  • 35