1

I am building an application that uses Jclouds (version 2.1.0) to access Google Cloud Storage and upload/download files. I am encountering an issue with multipart uploads. Not sure if I am doing something wrong, or if this is a bug.

What I am seeing is whenever I do a multipart upload, I get a "400 Bad Request" response. Here is the code I am using to upload: 

String fileContents = Files.toString(new File("google_creds.json"), Charset.defaultCharset());

Supplier<Credentials> credentialSupplier = new GoogleCredentialsFromJson(fileContents);

Iterable<Module> modules = ImmutableSet.<Module>of(
        new SLF4JLoggingModule());

BlobStoreContext context = ContextBuilder.newBuilder("google-cloud-storage")
        .credentialsSupplier(credentialSupplier)
        .modules(modules)
        .buildView(BlobStoreContext.class);

BlobStore client = context.getBlobStore();

byte[] testBytes = new byte[102400];

Blob blob = client.blobBuilder("testupload").payload(testBytes)
        .contentLength(testBytes.length).build();


// Multipart upload fails
 client.putBlob("filecatalyst-unit-tests", blob, multipart());

And here is the logging output that I see:

>> "{"sourceObjects":[{"name":"3fda0043-e340-4ccb-a582-9e3b78cf85e1_00000001","generation":1520026856033315,"objectPreconditions":{"ifGenerationMatch":1520026856033315}}],"destination":{"name":"testupload","size":102400,"contentType":"application/unknown","storageClass":"STANDARD","metadata":{}}}"
>> POST https://www.googleapis.com/storage/v1/b/filecatalyst-unit-tests/o/testupload/compose HTTP/1.1
>> Accept: application/json
>> Authorization: Bearer ***********************
>> Content-Type: application/json
>> Content-Length: 293
<< HTTP/1.1 400 Bad Request
<< Alt-Svc: hq=":443"; ma=2592000; quic=51303431; quic=51303339; quic=51303338; quic=51303337; quic=51303335,quic=":443"; ma=2592000; v="41,39,38,37,35"
<< Server: UploadServer
<< X-GUploader-UploadID: AEnB2UpqKslm87V4wto81YsNphtPFtybHjWgffGAC3XPE-xQAXSjDldst5s5nuWPyhXjEGOwWE5kDETjNA1EedbuEw9Vbe8CtA
<< Vary: X-Origin
<< Vary: Origin
<< Date: Fri, 02 Mar 2018 21:40:56 GMT
<< Content-Type: application/json; charset=UTF-8
<< Content-Length: 178
<< "{[\n]"
<< " "error": {[\n]"
<< "  "errors": [[\n]"
<< "   {[\n]"
<< "    "domain": "global",[\n]"
<< "    "reason": "invalid",[\n]"
<< "    "message": "Invalid argument"[\n]"
<< "   }[\n]"
<< "  ],[\n]"
<< "  "code": 400,[\n]"
<< "  "message": "Invalid argument"[\n]"
<< " }[\n]"
<< "}[\n]"

Here is a formatted version of the JSON body for the compose request:

{  
   "sourceObjects":[  
      {  
         "name":"83116375-0ba1-4d1a-aec3-8dd29dab0da9_00000001",
         "generation":1520263556434885,
         "objectPreconditions":{  
            "ifGenerationMatch":1520263556434885
         }
      }
   ],
   "destination":{  
      "name":"testupload",
      "size":33554432,
      "contentType":"application/unknown",
      "storageClass":"STANDARD",
      "metadata":{  

      }
   }
}

Have tried with varying size files, and I get the same result. If I do not use multipart upload, it works fine.

Andrew Gaul
  • 2,296
  • 1
  • 12
  • 19
Chris
  • 55
  • 1
  • 5
  • Can you see the request body? Hard to tell what's wrong without it. – Brandon Yarbrough Mar 03 '18 at 00:23
  • I believe that the first line of the log I included is the body of the HTTP request. Not sure why but Jclouds always logs the body before the headers. – Chris Mar 05 '18 at 15:33
  • I should add that the upload of the part/file "3fda0043-e340-4ccb-a582-9e3b78cf85e1_00000001" being referenced in the compose call was uploaded successfully, and I can see it in the GCP storage browser. Its just when it comes to trying to complete the multipart upload by creating the composite object that it is having problems. – Chris Mar 05 '18 at 15:42
  • Odd. Offhand, I'm not sure what's wrong. Could you maybe try composing the objects using a known-good method like gsutil and seeing if the call succeeds? – Brandon Yarbrough Mar 05 '18 at 20:22
  • gsutil successfully completes the upload, so certainly seems to be an issue with Jclouds. I ended up working around the issue by bypassing Jclouds for the completion of the multipart upload. I used the Google Cloud Storage API for Java for that one call, and it is able to successfully create the composite object using the parts uploaded using jclouds. Not ideal, as I use jclouds to access several other object stores and its nice to only maintain one set of code, but up an running. – Chris Mar 07 '18 at 14:41
  • I successfully ran the same code against GCS with the original 102400 input and with the logged 33554432. Does your test have some other state, e.g., overwriting an object? – Andrew Gaul Apr 07 '18 at 20:35
  • Looks like the problem was something to do with the storage class/region. So it would work depending on how your container is set up. I had logged the issue in the Jclouds JIRA, and it has some traction now so you can follow the issue https://issues.apache.org/jira/browse/JCLOUDS-1389?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&focusedCommentId=16446071#comment-16446071 – Chris Apr 20 '18 at 18:59

1 Answers1

1

JCLOUDS-1389 tracks this issue and jclouds 2.1.1 and 2.2.0 will include the fix.

Andrew Gaul
  • 2,296
  • 1
  • 12
  • 19