0

Environment:
Java client ("google-api-services-storage", "v1-rev33-1.20.0") using JSON API (com.google.api.services.storage.Storage class).

Goal:
Move a large object from "standard" to "nearline" bucket using Java client (file size is 512 MB).

Steps:
Use "rewrite" API method.

Problem:
I'm getting SocketTimeoutException in 20 seconds.

Investigation:
The same code works fine when I use rewrite from "standard" bucket to another "standard" bucket for the same object.
I've also tried the APIs Explorer and created a request to rewrite an object from "standard" to "nearline" bucket. The server responded in about 27 seconds and "totalBytesRewritten" property in the response was for about a half of file size. How to get and handle such response?
Documentation says:
"If the source and destination are different locations and/or storage classes, the rewrite method might require multiple calls."

My code (Java):

final Storage.Objects.Rewrite rewriteRequest = storage.objects().rewrite(
       STANDARD_BUCKET_NAME,
       SOURCE_OBJECT_PATH,
       NEARLINE_BUCKET_NAME,
       TARGET_OBJECT_PATH,
       null // no metadata overriding
);

rewriteRequest.execute();

Please help.

Alexander Litus
  • 25
  • 5
  • Can you post your code snippet? – jterrace May 12 '15 at 16:05
  • How many seconds does it take for you to get the SocketTimeoutException? Rewrites calls can last a while, but I don't believe it's longer than the default socket timeout for the Google API Java client default timeout. The API explorer results, though, are correct. – Brandon Yarbrough May 13 '15 at 01:01

1 Answers1

0

According to the documentation, if a file was split up into chunks, you are supposed to call rewrite again, using the 'rewriteToken' returned in the first response from rewrite. The operation will resume, doing one more chunk of data. This should be repeated until the response has getDone() == true.

My implementation for the java api:

private void rewriteUntilDone(final String sourceBucket, final String sourceKey,
                              final String destBucket,  final String destKey) throws IOException {
    rewriteUntilDone(sourceBucket, sourceKey, destBucket, destKey, null);
}

private void rewriteUntilDone(final String sourceBucket, final String sourceKey,
                              final String destBucket, final String destKey,
                              @Nullable final String rewriteToken)
        throws IOException {

    Storage.Objects.Rewrite rewrite = googleStorage.objects().rewrite(sourceBucket, sourceKey, destBucket, destKey, null);
    if (rewriteToken != null) {
        rewrite.setRewriteToken(rewriteToken);
    }
    RewriteResponse rewriteResponse = rewrite.execute();

    if (!rewriteResponse.getDone()) {
        String rewriteToken2 = rewriteResponse.getRewriteToken();
        BigInteger totalBytesRewritten = rewriteResponse.getTotalBytesRewritten();
        log.debug("Rewriting not finished, bytes completed: {}. Calling rewrite again with token {}", totalBytesRewritten, rewriteToken2);
        rewriteUntilDone(sourceBucket, sourceKey, destBucket, destKey, rewriteToken2);
    }
}

EDIT: Also, you may have to increase the read timeout. It seems that rewrite responds after 27 s but the default timout is 20 s. Wrap your GoogleCredentials to set the read timeout

Johan Tidén
  • 664
  • 7
  • 19