4

I am working on a project to upload objects to S3 using java code. There are some external restrictions that limit my implementation and overall I'm not sure if S3 supports what I'm trying to do.

The restrictions are:

  • Use V4 authentication
  • header authentication, not query parameter
  • REST API, not AWS java SDK
  • Payload is not hashed (no SHA-256)

That last requirement is because we have hardware support that streams the data directly from storage, so the driving code never touches the data.

Apparently with query parameter authentication I can substitute 'UNSIGNED-PAYLOAD' for the payload hash, but not so with header based authentication.

So my question is whether or not there is any way to upload an object to S3 using the REST API, v4 signature and no hash (SHA-256 or other) on the data itself.

Thanks!

Joe Meadows
  • 61
  • 2
  • 6
  • 1
    Even if possible, it sounds like a bad idea, since you have no way of confirming that the upload arrived intact. It's easy enough, if you can stream the data through your code, to do a multipart upload, which only requires you to provide md5 checksums that you can calculate on the fly for each uploaded "part," without knowing the size of the stream in advance. – Michael - sqlbot Oct 02 '14 at 04:25

2 Answers2

1

No, according to this post on Amazon's forums:

Re: https://forums.aws.amazon.com/message.jspa?messageID=573632

UNSIGNED-PAYLOAD can be used only with a query-string authentication. If you use Authorization header authentication, it cannot be used. As an option, you can use chunked transfer, so will have to calculate hashes for small chunks of data than can be buffered for hashing. Also, you can still use older Signature V2 , though it won't work with regions created after 30-jan-2014.

It looks like you can do this with v2 signatures using the header method but, as mentioned above, only to endpoints created before Jan 30th, 2014.

See: http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html#RESTAuthenticationStringToSign

jcoffland
  • 5,238
  • 38
  • 43
  • 3
    While I don't know when it changed, I don't believe this is true any longer - `UNSIGNED-PAYLOAD` is permitted with the Authorization header method of authentication: https://docs.amazonaws.cn/en_us/AmazonS3/latest/API/sig-v4-header-based-auth.html – James Addison Jan 14 '19 at 20:44
0

You can upload files using POST and it does not require payload hash. But with POST file size is limited to 5GB.

http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authentication-HTTPPOST.html

Seunghoon
  • 5,632
  • 5
  • 35
  • 41