I'm trying to get a large object from S3 and steam the output straight to an HttpServletResponse output stream.
However it always fails on this object due to a SocketTimeoutException.
Here's the code that uses the S3Client to get the object:
public void downloadMergedResults(final String jobId, final OutputStream os) {
get("jobs/" + jobId + "/results", os);
}
private boolean get(final String key, final OutputStream outputStream) {
log.trace("GET {}", key);
try {
client.getObject(GetObjectRequest.builder().bucket(bucket).key(key).build(), ResponseTransformer.toOutputStream(outputStream));
return true;
} catch (S3Exception e) {
if (e.statusCode() == 404) {
log.info("Item with key: {} does not exist.", key);
}
else {
log.error("Could not get item {} from S3. Error: {} {}", key, e.getMessage(), e.awsErrorDetails());
}
}
return false;
}
Here's how I'm using that within my springboot application:
public Callable<ResponseEntity<StreamingResponseBody>> allData(HttpServletResponse res, @PathVariable("jobId") String jobId) {
Progress progress = s3client.downloadCachedProgress(jobId);
if (progress == null) {
log.info("{} Could not find progress file", jobId);
return () -> new ResponseEntity<>(HttpStatus.NOT_FOUND);
}
res.setContentType(MediaType.APPLICATION_JSON_VALUE);
res.setContentLengthLong(OPEN_ARRAY.length + progress.getDataSizeBytes() + (SEPARATOR.length * (progress.getMediaCount() - 1)) + CLOSE_ARRAY.length);
return () -> {
StreamingResponseBody streamingResponseBody = out -> {
try {
s3client.downloadMergedResults(jobId, out);
} catch(Exception e) {
log.error("{} Error downloading results: {}", jobId, e.getMessage(), e);
}
};
return new ResponseEntity<>(streamingResponseBody, HttpStatus.OK);
};
}
This is what curl says:
curl: (18) transfer closed with 524076376 bytes remaining to read
This is the stack trace from my app:
software.amazon.awssdk.core.exception.SdkClientException: Unable to unmarshall response (java.net.SocketTimeoutException). Response Code: 200, Response Text: OK
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:108) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:72) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:189) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:121) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:147) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:62) ~[aws-core-2.13.44.jar!/:na]
at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:3539) ~[s3-2.13.44.jar!/:na]
at org.me.blobstorage.S3ClientWrapper.get(S3ClientWrapper.java:362) ~[classes!/:na]
at org.me.blobstorage.S3ClientWrapper.downloadMergedResults(S3ClientWrapper.java:253) ~[classes!/:na]
at org.me.RequestHandler.lambda$null$4(RequestHandler.java:241) ~[classes!/:na]
at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:111) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:98) ~[spring-webmvc-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at org.springframework.web.context.request.async.WebAsyncManager.lambda$startCallableProcessing$4(WebAsyncManager.java:328) ~[spring-web-5.2.2.RELEASE.jar!/:5.2.2.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_265]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_265]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_265]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_265]
Caused by: software.amazon.awssdk.core.exception.NonRetryableException: java.net.SocketTimeoutException
at software.amazon.awssdk.core.exception.NonRetryableException$BuilderImpl.build(NonRetryableException.java:92) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.transformResponse(BaseSyncClientHandler.java:187) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.handle(BaseSyncClientHandler.java:168) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleSuccessResponse(CombinedResponseHandler.java:97) ~[sdk-core-2.13.44.jar!/:na]
... 40 common frames omitted
Caused by: org.apache.catalina.connector.ClientAbortException: java.net.SocketTimeoutException
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:351) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:776) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:681) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:386) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:364) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at software.amazon.awssdk.utils.IoUtils.copy(IoUtils.java:113) ~[utils-2.13.44.jar!/:na]
at software.amazon.awssdk.utils.IoUtils.copy(IoUtils.java:99) ~[utils-2.13.44.jar!/:na]
at software.amazon.awssdk.core.sync.ResponseTransformer.lambda$toOutputStream$2(ResponseTransformer.java:159) ~[sdk-core-2.13.44.jar!/:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler$HttpResponseHandlerAdapter.transformResponse(BaseSyncClientHandler.java:180) ~[sdk-core-2.13.44.jar!/:na]
... 42 common frames omitted
Caused by: java.net.SocketTimeoutException: null
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:134) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1257) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:814) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:584) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:522) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:538) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:73) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:190) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.coyote.Response.doWrite(Response.java:601) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:339) ~[tomcat-embed-core-9.0.29.jar!/:9.0.29]
... 51 common frames omitted
Here are the different ways I've tried initialising my S3Client:
1:
client = S3Client.builder()
.region(region)
.credentialsProvider(credentialsProvider)
.build();
2:
client = S3Client.builder()
.region(region)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofHours(1))
.build())
.credentialsProvider(credentialsProvider)
.build();
3:
client = S3Client.builder()
.region(region)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofHours(1))
.build())
.httpClient(UrlConnectionHttpClient.builder()
.connectionTimeout(Duration.ofHours(1))
.socketTimeout(Duration.ofHours(1)).build())
.credentialsProvider(credentialsProvider)
.build();
4:
client = S3Client.builder()
.region(region)
.overrideConfiguration(ClientOverrideConfiguration.builder()
.apiCallTimeout(Duration.ofHours(1))
.build())
.httpClientBuilder(UrlConnectionHttpClient.builder()
.connectionTimeout(Duration.ofHours(1))
.socketTimeout(Duration.ofHours(1)))
.credentialsProvider(credentialsProvider)
.build();
My spring boot applications.properties setting for request timeouts:
spring.mvc.async.request-timeout=-1
and I have @EnableAsync on my bootstrap class.
The requests are proxied through nginx, with the settings on the location as:
upstream myupstream {
server [SERVER-REDACTED]:[PORT-REDACTED];
keepalive 15;
}
server {
...
location ~ ^/(v1/bigsearch|actuator)/ {
proxy_pass http://myupstream;
proxy_http_version 1.1;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
client_max_body_size 2G;
}
...
}
And the relevant nginx.conf entries itself:
http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 1d;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
proxy_read_timeout 1d;
proxy_send_timeout 1d;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
gzip on;
gzip_disable "msie6";
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
Question
What haven't I done to make this work?
