0

Can I use FlatfileItemReader with Taskexecutor in spring batch??

I have implemented FlatFileItemReader with ThreadPoolTaskExecutor. When I print the records in ItemProcessor, I do not get consistent results, i.e. not all the records are printed and sometimes one of the record is printed more than once. It leads me to the fact that FlatFileItemReader is not thread safe and also it says the same in spring docs but I see some blogs where it says it is possible to use FlatFileItemReader with Task Executor.

So my question is: Is it possible to use FlatfileItemReader with Task Executor in anyway ?

    @Bean
    @StepScope
    public FlatFileItemReader<DataLifeCycleEvent> csvFileReader(
            @Value("#{stepExecution}") StepExecution stepExecution) {

        Resource inputResource;
        FlatFileItemReader<DataLifeCycleEvent> itemReader = new FlatFileItemReader<>();

        itemReader.setLineMapper(new OnboardingLineMapper(stepExecution));
        itemReader.setLinesToSkip(1);
        itemReader.setSaveState(false);
        itemReader.setSkippedLinesCallback(new OnboardingHeaderMapper(stepExecution));
        String inputResourceString = stepExecution.getJobParameters().getString("inputResource");
        inputResource = new FileSystemResource(inputFileLocation + ApplicationConstant.SLASH + inputResourceString);
        itemReader.setResource(inputResource);
        stepExecution.getJobExecution().getExecutionContext().putInt(ApplicationConstant.ERROR_COUNT, 0);
        return itemReader;
    }
Prateek Gupta
  • 2,422
  • 2
  • 16
  • 30
Ritesh
  • 25
  • 5
  • Why would you need that? Do yo mean using a task executor with a chunk-oriented step? Please explain what are you trying to achieve without referring to Spring Batch to understand your requirement. – Mahmoud Ben Hassine May 26 '20 at 07:41
  • Hi @MahmoudBenHassine , my requirement is I want to upload a flat file in multiple threads like each thread takes set of 100 records ( for example ) , validates the record , in case if validation is success, write it to database else write it to error file. – Ritesh May 26 '20 at 15:40
  • In terms of spring batch i have implemented this with FlatFileItemReader to read the file and add task executor in my step to enable multi threading. When i print the records in item processor, it gives inconsistent results each time, some records are skipped and others are printed duplicate. – Ritesh May 26 '20 at 15:58

2 Answers2

1

FlatFileItemReader extends AbstractItemCountingItemStreamItemReader which is NOT thread-safe. So if you use it in a multi-threaded step, you need to synchronize it.

You can wrap it in a SynchronizedItemStreamReader. Here is a quick example:

@Bean
public SynchronizedItemStreamReader<DataLifeCycleEvent> itemReader() {
    FlatFileItemReader<DataLifeCycleEvent> itemReader = ... // your item reader

    SynchronizedItemStreamReader<DataLifeCycleEvent> synchronizedItemStreamReader = new SynchronizedItemStreamReader<>();
    synchronizedItemStreamReader.setDelegate(itemReader);
    return synchronizedItemStreamReader;
}
Mahmoud Ben Hassine
  • 28,519
  • 3
  • 32
  • 50
  • Thanks @Mahmoud Ben Hasine. This solved the issue. To decide weather i should send the record to Error file writer or Database writer , I implemented ClassifyCompositeItemWriter but sonar throw error that both the writers used inside ClassifyCompositeItemWriter should also be Serialized and all other objects inside those writers. I have faced this issue for the first time in Spring only with ClassifyCompositeItemWriter which lead me to change my complete implementation and move to ItemStreamWriter. Do you have any solution to this problem ? – Ritesh May 26 '20 at 21:09
0

This method is giving this exception :-java.lang.ClassCastException: com.sun.proxy.$Proxy344 cannot be cast to org.springframework.batch.item.support.SynchronizedItemStreamReader

@Bean
public SynchronizedItemStreamReader<DataLifeCycleEvent> itemReader() {
    FlatFileItemReader<DataLifeCycleEvent> itemReader = ... // your item reader

    SynchronizedItemStreamReader<DataLifeCycleEvent> synchronizedItemStreamReader = new SynchronizedItemStreamReader<>();
    synchronizedItemStreamReader.setDelegate(itemReader);
    return synchronizedItemStreamReader;
}
Shivangi Shukla
  • 21
  • 2