6

I'm using AWS SDK for .NET and I was looking for a method to let user upload directly to a s3 storage.

I've come across two different ways offedered by aws:

Browser based upload: https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authentication-HTTPPOST.html

and presigned urls: https://docs.aws.amazon.com/AmazonS3/latest/dev/UploadObjectPreSignedURLDotNetSDK.html

It seems presigned url is 'easier' since a method is already present within aws sdk to generate a url to pass to client, to let him PUT object directly to the bucket (and it seems also painless compared to browser upload, since it doesn't require all the keys browser upload wants in the post form).

But I was wondering why there are actually two different methods. What are PRO and CONS of each one?

nicecatch
  • 1,687
  • 2
  • 21
  • 38

2 Answers2

5

It totally comes down to the point, whether you want to use the REST API or the AWS SDKs to interact with S3.

In both cases, you need to prove(authenticate/Sign-Request) your identity unless bucket is public.

a) If you are going with REST APIs, to prove identity, you need sign your request using ' AWS Signature version 4 ' (deprecated ver 2 is also there), which includes three methods (one you have listed)

  1. Authenticating Requests: Using the Authorization Header (AWS Signature Version 4)
  2. Authenticating Requests: Using Query Parameters (AWS Signature Version 4)
  3. Authenticating Requests: Browser-Based Uploads Using POST (AWS Signature Version 4)

b) If you are going to use AWS SDKs, you should let SDK do the signing ceremony(process). So the choice is straightforward to use SDK to sign the request

(Part of the question) It seems also painless compared to browser upload since it doesn't require all the keys browser upload wants in the post form<

For below code, s3Client already has got your creds whether from AWS-CLI-Profile(if running local/laptop), IAM Role(in case of EC2, lambda, etc)

string url = s3Client.GetPreSignedURL(request);
raevilman
  • 3,169
  • 2
  • 17
  • 29
  • So if I am already using the .net sdk the presignedurl is the way to go? Can I set a max file size with presignedurl? – nicecatch Apr 04 '18 at 08:30
  • way to go> => yes. max file size => https://stackoverflow.com/questions/25991275/limit-size-of-objects-while-uploading-to-amazon-s3-using-pre-signed-url – raevilman Apr 04 '18 at 08:46
  • So there is no pre uplaod method with signed urls. I was already planning to use lambda to resize images uploaded, so I think I'll stick with it adding some sanity check (I suppose I have to check they are real images and no viruses or something similar) – nicecatch Apr 04 '18 at 11:05
1

The main difference that I have found is memory consumption. If you upload the same image (46KB) 200 times (as a test/benchmark) the memory consumption differs wildly.

PutObject with any version of AWSSDK.Core below 3.3.21.19 takes 116MB (SOH: 66MB / LOH: 50MB)

PutObject with any version of AWSSDK.Core above 3.3.21.19 takes 99MB (SOH: 98MB / LOH: 0.4MB)

You can further reduce the memory consumption if you set UseChunkEncoding to false on PutObjectRequest:-

PutObject with UseChunkEncoding set to false takes 52MB (SOH: 52MB / LOH: 0.4MB)

But GetPreSignedURL is still better from a memory point of view;

GetPreSignedURL takes 32MB (SOH: 32MB / LOH: 0.4MB)

If you are calling PutObject a lot then it may be benefical to switch out to GetPreSignedURL to save memory if that's a problem. The downside is that you are responsible for retrying mechanisms etc.

We use PutObject everywhere in our code base except one place where the code path is very hot and is hit around 800,000+ times a day. In that case it made sense to switch it out to GetPreSignedURL.

Hope that helps!

  • GetPreSignedURL is definitley better than user uploading to your web server and then putting object to s3 manually, saving bandwith, memory and cpu usage. I was asking myself which way to use between presignedurls and Browser-Based Uploads. It seems presignedurl are managed by the sdk (which I am already using), so I'll probably use them – nicecatch Apr 04 '18 at 11:08