2

We are using Chilkat 9.5.0.80 C++ library.

There is a certain HTTP header we cannot add to our requests: "Content-MD5". When we add this header like this:

m_ckRest.AddHeader("Content-MD5", "any-value-here");

and examine the resulting request*, the "Content-MD5" header is NOT present.

However, when we add a header of a different name:

m_ckRest.AddHeader("Content-Type", "application/octet-stream");

... the resulting request DOES contain that header. We are using the "fullRequestBinary" method, for example:

const char* responseStrPtr = m_ckRest.fullRequestBinary( "PUT", encodedObjectName.c_str(), ckByteDataBuffer);

* We are examining our requests using a Proxy (using "Fiddler" as an http proxy in between us and Amazon S3 for example to test the upload of a "part" in a multipart AWS S3 upload) and in every attempt, the "Content-MD5" header is NOT present, while other headers are present.

Is this a bug? We found an old forum post from 2013 referencing a very similar sounding problem: http://www.chilkatforum.com/questions/2901/addheader-range-does-not-appear-to-be-effective Does Chilkat remove or ignore our attempt to add a "Content-MD5" header? Is this bug fixed in a version newer than the one we are using? Is there a workaround? Here is an example of the headers in a PUT request:

PUT https://our-bucket.s3.us-west-1.amazonaws.com/somefile?partNumber=4&uploadId=tJJYIXdxG_7X8elzSJrKt32A_rH46Y0Yk1vyzZgwxpvmK5uCrcE82k_F9UmytVHWuxXfc6tX5o3w.SRnnYcD7VBskcLrr0xC13bHHVDx62iGGQ3eIzkv5J5d1F4_DkcW HTTP/1.1
Content-Length: 5266235
x-amz-date: 20200921T201943Z
x-amz-content-sha256: 90fa8fc564dd558d0c2eac92e367d94101f4ca9570c970795b9fdb2aa96d6666
Host: our-bucket.s3.us-west-1.amazonaws.com
Content-Type: application/octet-stream
Date: Mon, 21 Sep 2020 20:19:43 GMT
Authorization: AWS4-HMAC-SHA256 Credential=AKIAIBYS55OSD2FIOBFUS/20200921/us-west-1/s3/aws4_request,SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date,Signature=8ea74cb7769d8e158e5ccc0604cc2cdb096703b10c3c8d9323d0746debbdUUU
AdamE
  • 606
  • 6
  • 11

2 Answers2

1

In correspondence with Chilkat support, turns out that Chilkat versions 9.5.0.80 and 9.5.0.83 intentionally remove the Content-MD5 header when authenticating using AWS Signature V4. Instead, Chilkat calculates the SHA256 hash and places it in the x-amz-content-sha256 (if authenticating using older AWS signature V2, it calculates Content-MD5 I'm told) So, unlike the comment from @Chilkat Software, this has not been fixed in a later version as of the writing of that comment, and the removal is intentional.

This is not terrible, but it stems from the misunderstanding that a SHA-256 hash of the content is necessary to construct a valid AWS Signature V4 for authentication, when in fact that is not the case. While SHA256 is perfectly adequate for content verification, it is also wasteful in comparison to MD5 for content verification.

The AWS C++ SDK itself does not use a SHA-256 hash in the x-amz-content-sha256 header when uploading a part. I've confirmed it uses: x-amz-content-sha256:UNSIGNED-PAYLOAD and instead uses the less "costly" MD5 hash, and puts it in the Content-MD5 header (see AWS documentation here https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html)

Unsigned payload option – You include the literal string UNSIGNED-PAYLOAD when constructing a canonical request, and set the same value as the x-amz-content-sha256 header value when sending the request to Amazon S3

Here is an example of an Amazon AWS UploadPart request using Content-MD5 for conent verification, and NOT using SHA256 for signing the request (captured from request using AWS SDK for C++):

PUT https://mybucket.s3.us-west-1.amazonaws.com/somefile.mfs01?partNumber=1&uploadId=6CHL6tPKFcRSoxD4iysjKMgQCNfcFAt87bn4fsduV1YI5_aFIz9e36BxFURH_iEX8EChUtQm06qT9oyIUDbAnA.2M.novpBBKsnGl_NqNvVllQ7L1VK6x1PiLlqq46tH HTTP/1.1
Cache-Control: no-cache
Connection: Keep-Alive
Pragma: no-cache
Content-Type: binary/octet-stream
Content-MD5: PV204S0m8zJY8zu9Q3EF+w==
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=AKIAIBYS55OSD2FOBFUSC/20200923/us-west-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-md5;content-type;host;x-amz-content-sha256;x-amz-date, Signature=d013028d77e45f3dcce5f46f3fb53cdeeb3c9cfbd931371e69a9925047e61cd3
Host: nuix-nov-dev.s3.us-west-1.amazonaws.com
User-Agent: aws-sdk-cpp/1.7.333 Windows/10.0.19041.329 x86 MSVC/1927
amz-sdk-invocation-id: E57D09A7-B5E7-4E2A-8B2D-B493147F06D7
amz-sdk-request: attempt=1
x-amz-content-sha256: UNSIGNED-PAYLOAD
x-amz-date: 20200923T212738Z
Content-Length: 5242880

Chilkat gave us a new "beta" build that allows us to specify the Content-MD5 header even for AWS Signature V4 and it won't remove it, however, it's in addition to the automatically calculated SHA-256 x-amz-content-sha256 so that's doubling the hashing unnecessarily, and would be better to be able to specify UNSIGNED-PAYLOAD for purposes of the AWS signature.

If there is a content mismatch error with the Content-MD5 value, AWS returns this (with a status 400):

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>InvalidDigest</Code>
    <Message>The Content-MD5 you specified was invalid.</Message>
    <Content-MD5>thisisbad</Content-MD5>
    <RequestId>8274DC9566D4AAA8</RequestId>
    <HostId>H6kSy4cl+54nMon1Hq6AGjmTX/MfTVMQQr8vEVNXUnPlfMtIt8HPdObfusckhBpwpG/CJ6ORWv16c=</HostId>
</Error>

If there is a content mismatch with x-amz-content-sha256 AWS returns the following error, which I had difficulty finding on the web, and is slightly different so pasting here (also status 400):

Status:400 : AWSCode: XAmzContentSHA256Mismatch : AWSMessage: The provided 'x-amz-content-sha256' header does not match what was computed.
AdamE
  • 606
  • 6
  • 11
0

This problem should have already been fixed in a later version of Chilkat.

Chilkat Software
  • 1,405
  • 1
  • 9
  • 8