How to specify the taskexecutor to spawn thread after reading a single csv by using MultiResourceItemReader in spring batch

Question

Trying to read multiple files in spring batch using MultiResourceItemReader and also have taskExecutor for the records in each file to read in multithreading. Suppose there are 3 csv files in a folder, MultiResourceItemReader should execute it one by one but as I have taskExecutor , different threads takes up csv files like two csv files are taken up by the threads from the same folder and starts executing .

Expectation:- MultiResourceItemReader should read first file and then taskExecutor should spawn different threads and execute. Then another file should be picked up and should be taken by taskExecutor for execution.

Code Snippet/Batch_Configuration :- @Bean public Step Step1() { return stepBuilderFactory.get("Step1") .<POJO, POJO>chunk(5) .reader(multiResourceItemReader()) .writer(writer()) .taskExecutor(taskExecutor()).throttleLimit(throttleLimit) .build(); }

@Bean
public MultiResourceItemReader<POJO> multiResourceItemReader() 
{
    MultiResourceItemReader<POJO> resourceItemReader = new MultiResourceItemReader<POJO>();
    ClassLoader cl = this.getClass().getClassLoader();
    ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(cl);

    Resource[] resources;
    try {
        resources = resolver.getResources("file:/temp/*.csv");
         resourceItemReader.setResources(resources);
            resourceItemReader.setDelegate(itemReader());
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    return resourceItemReader;
}

score 0 · Answer 1 · answered Jul 30 '20 at 19:13

maybe you should check Partitioner (https://docs.spring.io/spring-batch/docs/current/reference/html/scalability.html#partitioning) and implement something like the following:

@Bean
public Step mainStep(StepBuilderFactory stepBuilderFactory,
                     FlatFileItemReader itemReader,
                     ListDelegateWriter listDelegateWriter,
                     BatchProperties batchProperties) {
    return stepBuilderFactory.get(Steps.MAIN)
            .<POJO, POJO>chunk(pageSize)
            .reader(unmatchedItemReader)
            .writer(listDelegateWriter)
            .build();
}

@Bean
public TaskExecutor jobTaskExecutor(@Value("${batch.config.core-pool-size}") Integer corePoolSize,
                                    @Value("${batch.config.max-pool-size}") Integer maxPoolSize) {
    ThreadPoolTaskExecutor taskExecutor = new ThreadPoolTaskExecutor();
    taskExecutor.setCorePoolSize(corePoolSize);
    taskExecutor.setMaxPoolSize(maxPoolSize);
    taskExecutor.afterPropertiesSet();
    return taskExecutor;
}

@Bean
@StepScope
public Partitioner partitioner(@Value("#{jobExecutionContext[ResourcesToRead]}") String[] resourcePaths,
                               @Value("${batch.config.grid-size}") Integer gridSize) {
    Resource[] resourceList = Arrays.stream(resourcePaths)
            .map(FileSystemResource::new)
            .toArray(Resource[]::new);
    MultiResourcePartitioner partitioner = new MultiResourcePartitioner();
    partitioner.setResources(resourceList);
    partitioner.partition(gridSize);
    return partitioner;
}

@Bean
public Step masterStep(StepBuilderFactory stepBuilderFactory, Partitioner partitioner, Step mainStep, TaskExecutor jobTaskExecutor) {
    return stepBuilderFactory.get(BatchConstants.MASTER)
            .partitioner(mainStep)
            .partitioner(Steps.MAIN, partitioner)
            .taskExecutor(jobTaskExecutor)
            .build();
}

How to specify the taskexecutor to spawn thread after reading a single csv by using MultiResourceItemReader in spring batch

1 Answers1