0

I have an S3 bucket in a Cloudformation stack that holds artifacts. A lambda function in the same stack generates a presigned URL for clients to upload files into the bucket.

I already had the code working beforehand, but after I redeployed the stack today, none of my clients were able to upload anything to the presigned URLs – S3 seems to close the connection without returning a response. I checked the lambda function had CRUD access to the bucket – but to rule out IAM issues, I ran testing code (below) locally on my machine with admin IAM credentials:

import boto3
import requests

s3_client = boto3.client("s3")

# re-enacting what my lambda function would have done
data = s3_client.generate_presigned_post(Bucket="my-cursed-bucket", Key="somefile.txt")
# {'url' : 'https://my-cursed-bucket.s3.amazonaws.com', 'fields': ...}

# re-enacting what my REST clients would have done
file = open('somefile.txt', 'rb')
requests.post(url=data['url'], data=data['fields'], files={'file': file})

S3 never returns a response but closes it, and requests/urllib throws exceptions about broken pipes:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

I have never run into this issue before – when I had badly configured ACLs/presign conditions, S3 would at least return me a certain HTTP status code and an XML message saying what was wrong.

Frustratingly, this appears to be happening to only that specific bucket belonging to the stack. If I use the same code above to generate presigned URLs for any other bucket with a variety of ACL rules / CORS settings etc, they all work.

Redeploying the stack, or creating a copy of the stack under a different name (resulting in different bucket names, but still belonging to the stack) will still result in presigned uploads failing.

I had not made any changes to the bucket's properties in the stack when this issue started today. My bucket's declaration in my CloudFormation (SAM) template is simply:

MyCursedBucket:
    Type: AWS::S3::Bucket

and nothing more.

What is causing this issue? Is there a special setting needed for presign uploads to work that I left out in the CF template?

chesnutcase
  • 103
  • 1
  • 3
  • That's a weird one. Is there anything relevant in the S3 access logs or CloudTrail logs? You might have to configure CloudTrail to record S3 events - if you make that change give it 15 minutes to apply the change then it takes 15 minutes for the logs to appear in the logging S3 bucket. – Tim Jul 31 '20 at 18:40
  • I suspected it was something odd happening at AWS so I gave up and let it sit for 12 hours without making any changes. When I came back, it was mysteriously working again. If anyone in the future sees this, it's best to take a break and leave it be – I don't believe any of AWS API's would intentionally kill your request at the network connection level without sending an informative error response. Feel free to close if this is considered an unsolvable question. – chesnutcase Aug 01 '20 at 18:08
  • 1
    Weird. AWS isn't perfect though. Suggest you put that in as an answer, then accept your own answer 24 hours later. – Tim Aug 01 '20 at 21:36

0 Answers0