Streaming uploading to S3 using presigned URL

Question

I'm trying to upload a file into a customer's S3. I'm given a presigned URL that allows me to do a PUT request. I have no access to their access and secret key so the use of the AWS SDK is out of the question.

The use case is that I am consuming a gRPC server streaming call and transforming it into a csv with some field changes. As the calls come in, I would want to be able to stream the transformed gRPC response into S3. I would need to do it via streaming cause the response can get rather large, upwards of >100mb, so loading everything into memory before uploading it into S3 is not ideal. Any ideas?

Answer 1

This is an open issue with pre-signed S3 upload URL:

https://github.com/aws/aws-sdk-js/issues/1603

Currently, the only working solution for large upload through S3 pre-signed URL is to use multi-part upload. The biggest drawback of that approach is you need to let the server that signs the upload know the size of your file, as it will need to pre-sign each part individually. e.g: 100MB file upload will require 20 parts (each 5MB maximum) to be pre-signed individually.