Reading content from google client http response multiple times in a batch request

Question

Following the Google Drive REST API documentation, I'm implementing an exponential backoff strategy for recoverable API errors. I'd like to retry my request if the error code is 500 or 403 (and maybe 429), but I noticed that not all 403 errors are recoverable, so I'd like to retry only for those with a reason in a selected list. The error reason is not provided as a separate information in the response, but it's contained in the content inputstream, which I parsed and processed:

private boolean isRateLimitExceeded(com.google.api.client.http.HttpResponse response) {
    int statusCode = response.getStatusCode();
    if (statusCode == 403) {
        response.getRequest().setParser(Utils.getDefaultJsonFactory().createJsonObjectParser());
        GoogleJsonErrorContainer jsonResponse = response.parseAs(GoogleJsonErrorContainer.class);
        List<ErrorInfo> errors = jsonResponse.getError().getErrors();
        if (errors != null && !errors.isEmpty()) {
            List<String> recoverableErrors = Arrays.asList("limitExceeded", "concurrentLimitExceeded", "rateLimitExceeded", "userRateLimitExceeded", "servingLimitExceeded");
            for (ErrorInfo errorInfo : errors) {
                if (!recoverableErrors.contains(errorInfo.getReason())) {
                    return false;
                }
            }
            return true;
        }
    }
    return false;
}

This works fine, but consumes the content. During the execution of the batch request, the google client tries to read and parse the content again, finds nothing left, and throws an exception. I tried marking and resetting the response.getContent() inputstream, but Google's HttpResponse doesn't support marking. Is there any other way I can make the content available again, or reading it without consuming it?

If it matters, I'm using a GoogleNetHttpTransport.

rgrebski · Answer 1 · 2018-06-03T19:45:51.843

I believe this is the proper way to do that:

private static final String USER_RATE_LIMIT_EXCEEDED_REASON = "userRateLimitExceeded";
private static final String RATE_LIMIT_EXCEEDED_REASON = "rateLimitExceeded";
private static final JsonFactory JSON_FACTORY = JacksonFactory.getDefaultInstance();

private boolean isRateLimitExceeded(HttpResponse response) {
    return response.getStatusCode() == HttpStatusCodes.STATUS_CODE_FORBIDDEN
            && isRateLimitExceeded(GoogleJsonResponseException.from(JSON_FACTORY, response));
}

private boolean isRateLimitExceeded(GoogleJsonResponseException ex) {
    if (ex.getDetails() == null || CollectionUtils.isEmpty(ex.getDetails().getErrors())) {
        return false;
    }
    String reason = ex.getDetails().getErrors().get(0).getReason();
    return RATE_LIMIT_EXCEEDED_REASON.equals(reason) || USER_RATE_LIMIT_EXCEEDED_REASON.equals(reason);
}

See also: https://www.programcreek.com/java-api-examples/?code=cchabanois/mesfavoris/mesfavoris-master/bundles/mesfavoris.gdrive/src/mesfavoris/gdrive/connection/GDriveBackOffHttpRequestInitializer.java

That doesn't help - it still parses and consumes the response. — gbaso, Jun 11 '18 at 13:47
You may be doing something wrong, I have already implemented similar backoff mechanism only in case 403 rate limit and everything works just perfect. It would be easier if u pasted your code and exception here — rgrebski, Jun 12 '18 at 07:46
Maybe I was not clear enough, but the backoff does work. But at the end of the batch request, the program crashes because the response content was consumed (by the backoff), while it expected it to be readable. — gbaso, Jun 14 '18 at 08:34

Reading content from google client http response multiple times in a batch request

1 Answers1