1

I am trying to set the Content-MD5 header when I use Minio Golang SDK to upload a file to S3. I can successfully upload files to AWS without setting Content-MD5, but uploading to IBM Cloud Object Storage fails with the following error:

ERR: Object write failed, reason: Missing required header for this request: Content-MD5

According to the Minio SDK,
https://docs.minio.io/docs/golang-client-api-reference#FPutObject
I use the UserMetadata field in minio.PutObjectOptions to set Content-MD5, but IBM Cloud Object Storage keeps complaining missing MD5, am I doing something wrong in the following code?

func (cloudIO *CloudIO) FWrite(name string) (n int, err error) {
    f, err := os.Open(name)
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()

    h := md5.New()
    if _, err := io.Copy(h, f); err != nil {
        log.Fatal(err)
    }

    bytesWritten, err := cloudIO.client.FPutObject(cloudIO.bucket, cloudIO.address,
        name,
        minio.PutObjectOptions{UserMetadata: map[string]string{"Content-MD5": hex.EncodeToString(h.Sum(nil))}})
    return int(bytesWritten), err
}
pacalJ
  • 13
  • 1
  • 4
  • 1
    Looking at the sources, it looks to me this MD5 should be added automatically. Providing it as a custom header like you did, it will not make it to the final outgoing request, but the key will be prefixed by `"X-Amz-Meta-"` (as can be seen in [api-put-object.go / Header() method](https://github.com/minio/minio-go/blob/master/api-put-object.go#L91)). So to sum it, it's possible this is a bug, you should report it. – icza Jul 03 '18 at 18:48
  • @icza Thanks. I saw exactly the same thing in the go file you mentioned. I am trying to capture the packet to verify if Content-MD5 is missing in there. – pacalJ Jul 03 '18 at 19:27

2 Answers2

1

@pacalj If you look at AWS documentation for PutObject https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html . Content-MD5 is not a required field. Meaning if it is not set by the client, the server should not error out, as you have already seen with AWS S3. As far as minio-go sdk is concerned, content-MD5 cannot be set via PutObjectOptions as explained in https://docs.minio.io/docs/golang-client-api-reference#FPutObject

Minio-go sets X-Amz-Content-Sha256 in the case of http connection and Content-Md5 in the case of https connections. Minio-go's FPutObject and PutObject api abstracts both multi-part put and single part put into these APIs. In the case of multi-part PUT, each part will set either X-Amz-Content-Sha256 or Content-Md5 depending on the type of connection. Since the call is abstracted, it is not possible for a user to set Content-Md5.

I believe IBM Cloud Object Storage has a bug as it should not error out, even if Content-Md5 is not set.

r1j1m1n1
  • 345
  • 1
  • 4
  • thanks for the great answer. I figured out the same conclusion as Content-Md5 cannot be set by a user, it will be bypassed and not added to the header, hence causing the error. I modified Minio-go sdk to add compatibility for IBM COS and now it solves the problem. But I agree that it could be a bug in the IBM side. – pacalJ Jul 11 '18 at 17:10
0

You can also use minio Core instead of Client to set the MD5 and SHA256 headers. As the documentation says about Core:

NewCore - Returns new initialized a Core client, this CoreClient should be only used under special conditions such as need to access lower primitives and being able to use them to write your own wrappers.

It has a PutObject method that has the md5Base64 and sha256Hex arguments.

meshkati
  • 1,720
  • 2
  • 16
  • 29