Spring Batch - File to be read is not available

Question

I have a Spring batch job with a first step that reads a file and writes to a database. If the file does not exist, I want the system to wait x minutes and then retry the step up to y number of attempts. How can I do this?

Details

The ItemReader is as follows:

 @Bean
    public ItemReader<Car> reader(LineMapper<Car> lineMapper, File file) {
        FlatFileItemReader<Car> flatFileItemReader = new FlatFileItemReader<Car>();
        flatFileItemReader.setResource(new FileSystemResource(file));
        final int NUMBER_OF_HEADER_LINES = 1;
        flatFileItemReader.setLinesToSkip(NUMBER_OF_HEADER_LINES);
        flatFileItemReader.setLineMapper(lineMapper);
        return flatFileItemReader;
    }

Currently, if the file does not exist, the following exception is thrown and the job fails.

2016-08-05 09:30:10.277 ERROR 7668 --- [           main] o.s.batch.core.step.AbstractStep         : Encountered an error executing step stepLoadFile in job testJob

org.springframework.batch.item.ItemStreamException: Failed to initialize the reader
    at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:147) ~[spring-batch-infrastructure-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:96) ~[spring-batch-infrastructure-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.batch.core.step.tasklet.TaskletStep.open(TaskletStep.java:310) ~[spring-batch-core-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:197) ~[spring-batch-core-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) [spring-batch-core-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:392) [spring-batch-core-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:135) [spring-batch-core-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:306) [spring-batch-core-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135) [spring-batch-core-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.core.task.SyncTaskExecutor.execute(SyncTaskExecutor.java:50) [spring-core-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:128) [spring-batch-core-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_91]
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_91]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:1.8.0_91]
    at java.lang.reflect.Method.invoke(Unknown Source) ~[na:1.8.0_91]
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) [spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:127) [spring-batch-core-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
    at com.sun.proxy.$Proxy78.run(Unknown Source) [na:na]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.execute(JobLauncherCommandLineRunner.java:215) [spring-boot-autoconfigure-1.3.1.RELEASE.jar:1.3.1.RELEASE]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.executeLocalJobs(JobLauncherCommandLineRunner.java:232) [spring-boot-autoconfigure-1.3.1.RELEASE.jar:1.3.1.RELEASE]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.launchJobFromProperties(JobLauncherCommandLineRunner.java:124) [spring-boot-autoconfigure-1.3.1.RELEASE.jar:1.3.1.RELEASE]
    at org.springframework.boot.autoconfigure.batch.JobLauncherCommandLineRunner.run(JobLauncherCommandLineRunner.java:118) [spring-boot-autoconfigure-1.3.1.RELEASE.jar:1.3.1.RELEASE]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:804) [spring-boot-1.3.1.RELEASE.jar:1.3.1.RELEASE]
    at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:788) [spring-boot-1.3.1.RELEASE.jar:1.3.1.RELEASE]
    at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:775) [spring-boot-1.3.1.RELEASE.jar:1.3.1.RELEASE]
    at org.springframework.boot.SpringApplication.doRun(SpringApplication.java:366) [spring-boot-1.3.1.RELEASE.jar:1.3.1.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:305) [spring-boot-1.3.1.RELEASE.jar:1.3.1.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1124) [spring-boot-1.3.1.RELEASE.jar:1.3.1.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1113) [spring-boot-1.3.1.RELEASE.jar:1.3.1.RELEASE]
    at com.myorg.LoadApplication.main(LoadApplication.java:15) [bin/:na]
Caused by: java.lang.IllegalStateException: Input resource must exist (reader is in 'strict' mode): file [C:\test\TestData.csv]
    at org.springframework.batch.item.file.FlatFileItemReader.doOpen(FlatFileItemReader.java:251) ~[spring-batch-infrastructure-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:144) ~[spring-batch-infrastructure-3.0.6.RELEASE.jar:3.0.6.RELEASE]
    ... 33 common frames omitted

com.myorg.LoadApplication code starts the batch application and is:

@SpringBootApplication
public class LoadApplication {

    public static void main(String[] args) {
        SpringApplication.run(LoadApplication.class, args);
    }

}

Update

After reading dimitrisli answer, I tried this in my @Configuration class (Note: I also added @EnableRetry to the configuration class):

@Bean
public ItemReader<Test> reader(LineMapper<Test> lineMapper, ApplicationProperties properties) {
    FlatFileItemReader<Test> flatFileItemReader = new FlatFileItemReader<Test>() {
        @Override
        @Retryable(value = {ItemStreamException.class}, maxAttempts=5)
        public void open(ExecutionContext executionContext) throws ItemStreamException {
            super.open(executionContext);
        }

        @Override
        @Retryable(maxAttempts=5)
        public Holding read() throws UnexpectedInputException, ParseException, Exception {
            return super.read();
        }

    };
    flatFileItemReader.setResource(new FileSystemResource(properties.getLoadFile()));
    flatFileItemReader.setLineMapper(lineMapper);
    return flatFileItemReader;
}

ItemStreamException is thrown and the application exits without retrying. How can I get it to retry?

By applying dimitrisli's answer to the ItemReader (code shown in update above), I was able to get this to finally work. The application was exiting without retrying due apparently to a bug (?) in spring-boot-starter-batch version 1.3.1.RELEASE. After moving to 1.4.0.RELEASE, retry worked as expected. Marking dimitrisli's answer as correct as it answered my original question. — James, Aug 10 '16 at 19:02

score 4 · Accepted Answer · answered Aug 04 '16 at 21:28

4

By using Spring Retry, the standalone spinoff project from Spring Batch: Assuming the Exception potentially thrown, that you would want to recover from and retry, say 5 times, is ItemStreamException then you'll have to decorate your method that potentially throws this exception as:

@Retryable(value = {ItemStreamException.class}, maxAttempts = 5)
public void myMethod() {
   //..
}

answered Aug 04 '16 at 21:28

dimitrisli

20,895
12
59
63

The problem is that this exception is not being thrown from my code. Spring batch is throwing the exception. I've added full stack trace to the OP. Looking at the stack trace, I suppose I could add `@Retryable` main method (which is where the Spring boot batch application is run). Is that your recommendation or is there a better (more targeted) place to put this annotation? – James Aug 05 '16 at 14:37

UserF40 · Answer 2 · 2016-08-04T21:13:59.063

0

You should use the spring-retry .

A RetryTemplate in combination with a TimeoutRetryPolicy will allow you do write the feature you want.

You need to add the @EnableRetry to the configuration class and @Retryable to the method.

Is the ItemReader needed to be instantiated first before the application can be started? If not you should just create the bean and have another method initialize it after the context refreshes.

edited Aug 04 '16 at 21:13

answered Aug 04 '16 at 21:09

UserF40

3,533
2
23
34

I've read that link, but I'm not clear on how it can be used / configured for the scenario that I described. Can you please provide a code sample on how you would accomplish this? – James Aug 04 '16 at 21:13

Spring Batch - File to be read is not available

2 Answers2