I have a Spring Boot application with a scheduled task that uploads files to an S3 bucket using the AWS SDK for Java 2.13.10. On application shutdown I would like the scheduled task to finish uploading the current file and exit instead of moving on to the next file. However, after starting the application shutdown via SIGINT the AWS SDK immediately throws AbortedException (caused in some instances by SdkInterruptedException) resulting in the current upload failing and ungraceful termination of the scheduled task.
Here is a skeleton scheduled task that exhibits the same behavior:
@Component
public class TestScheduledTaskService implements ApplicationListener<ContextClosedEvent> {
public static final String AWS_ACCESS_KEY_ID = "";
public static final String AWS_ACCESS_KEY_SECRET = "";
public static final String AWS_BUCKET_NAME = "";
private static final Logger LOGGER = LoggerFactory.getLogger(TestScheduledTaskService.class);
private boolean shutdownStarted = false;
private S3Client s3Client;
public TestScheduledTaskService() {
AwsBasicCredentials credentials = AwsBasicCredentials.create(AWS_ACCESS_KEY_ID, AWS_ACCESS_KEY_SECRET);
s3Client = S3Client.builder()
.credentialsProvider(StaticCredentialsProvider.create(credentials))
.region(Region.EU_CENTRAL_1)
.build();
}
@Scheduled(fixedDelay = 10_0000)
public void scheduledTask() {
LOGGER.info("Starting scheduled task");
for (int i = 0; i < 10; i++) {
if (shutdownStarted) {
LOGGER.info("Application shutting down, stopping scheduled task");
return;
}
String key = String.valueOf(Instant.now().getEpochSecond());
LOGGER.info("Creating and uploading 5MiB test file with key {}", key);
byte[] randomBytes = new byte[1024 * 1024 * 5];
new Random().nextBytes(randomBytes);
PutObjectRequest putRequest = PutObjectRequest.builder()
.bucket(AWS_BUCKET_NAME)
.key(key)
.build();
s3Client.putObject(putRequest, RequestBody.fromBytes(randomBytes));
}
LOGGER.info("Scheduled task complete.");
}
@Override
public void onApplicationEvent(final ContextClosedEvent event) {
shutdownStarted = true;
LOGGER.info("Application is shutting down, stopping after next upload");
}
}
The Spring scheduler has been configured by defining a bean in a configuration class:
@Bean
public TaskSchedulerCustomizer taskSchedulerCustomizer() {
return taskScheduler -> {
taskScheduler.setAwaitTerminationSeconds(60);
taskScheduler.setWaitForTasksToCompleteOnShutdown(true);
};
}
The log output and complete stack trace on termination:
2020-05-07 00:44:56.384 INFO 144232 --- [extShutdownHook] c.e.d.service.TestScheduledTaskService : Application is shutting down, stopping after next upload
2020-05-07 00:44:56.386 INFO 144232 --- [extShutdownHook] o.s.s.c.ThreadPoolTaskScheduler : Shutting down ExecutorService 'taskScheduler'
2020-05-07 00:44:56.701 ERROR 144232 --- [ scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task
software.amazon.awssdk.core.exception.AbortedException: null
at software.amazon.awssdk.core.exception.AbortedException$BuilderImpl.build(AbortedException.java:84) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.io.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:45) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.io.SdkFilterInputStream.read(SdkFilterInputStream.java:65) ~[sdk-core-2.13.10.jar:na]
at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:140) ~[httpcore-4.4.13.jar:4.4.13]
at software.amazon.awssdk.http.apache.internal.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:149) ~[apache-client-2.13.10.jar:na]
at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.impl.conn.CPoolProxy.sendRequestEntity(CPoolProxy.java:152) ~[httpclient-4.5.12.jar:4.5.12]
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) ~[httpcore-4.4.13.jar:4.4.13]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272) ~[httpclient-4.5.12.jar:4.5.12]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.12.jar:4.5.12]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.12.jar:4.5.12]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.12.jar:4.5.12]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.12.jar:4.5.12]
at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) ~[apache-client-2.13.10.jar:na]
at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:232) ~[apache-client-2.13.10.jar:na]
at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:98) ~[apache-client-2.13.10.jar:na]
at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:213) ~[apache-client-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:66) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:51) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:35) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:189) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:121) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:147) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:101) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[sdk-core-2.13.10.jar:na]
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) ~[aws-core-2.13.10.jar:na]
at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:7376) ~[s3-2.13.10.jar:na]
at com.example.demo.service.TestScheduledTaskService.scheduledTask(TestScheduledTaskService.java:59) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:567) ~[na:na]
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.2.5.RELEASE.jar:5.2.5.RELEASE]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
at java.base/java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:305) ~[na:na]
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:830) ~[na:na]
Process finished with exit code 130 (interrupted by signal 2: SIGINT)
How can I prevent this behavior and allow the current file upload to complete?
There are many questions on Stack Overflow relating to graceful shutdown of Spring Boot applications but I have been unable to find any that are similar to the problem here. I believe the issue originates in the AWS client's handling of InterruptedException. Replacing the S3 file upload in the example with some other blocking operation results in graceful termination when starting the next loop iteration after the shutdown has been initiated without affecting the blocking operation.
One workaround I have considered is catching the exception thrown by the putObject call and attempting to delete a key of the same name to cleanup, but the call to deleteObject inside the catch block causes the exception to be rethrown. Ideally however I would like to avoid such cleanup on application shutdown if possible.
