How to read all files in a folder with spring-batch and MultiResourceItemReader?

Question

I want to configure spring-batch to read all csv files inside a specific folder sequentially.

The following does not work because the delegate will try to open a file named *.csv, which of course is invalid. What do I have to change here?

    @Bean
    public ItemReader<String> reader() {
        MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
        reader.setResources(new Resource[] {new FileSystemResource("/myfolder/*.csv")});
        reader.setDelegate(new FlatFileItemReader<>(..));
        return reader;
}

The equivalent xml configuration would be written as follows, how could I rewrite it to java only config?

<bean id="reader" class="org.springframework.batch.item.file.MultiResourceItemReader">
        <property name="resources" value="/mypfolder/*.csv"/>
        <property name="delegate" ref="flatFileItemReader"/>
</bean>

Funny that I came back to my own question right now, and marking this as a favorite :D — membersound, Mar 24 '16 at 09:49

score 16 · Answer 1 · edited Apr 23 '21 at 08:35

16

Use PathMatchingResourcePatternResolver like this.

@Bean
public ItemReader<String> reader() {
    Resource[] resources = null;
    ResourcePatternResolver patternResolver = new PathMatchingResourcePatternResolver();   
    try {
        resources = patternResolver.getResources("/myfolder/*.csv");
    } catch (IOException e) {
        e.printStackTrace();
    }
    MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
    reader.setResources(resources);
    reader.setDelegate(new FlatFileItemReader<>(..));
    return reader;
}

edited Apr 23 '21 at 08:35

EricSchaefer

25,272
21
67
103

answered Aug 17 '17 at 07:25

Tharindu Jayasuriya

364
3
9

Is there any way in this approach to print the skip,commit,read,write count per file? – Ashish Jun 22 '19 at 11:38

score 10 · Accepted Answer · answered Jul 29 '15 at 20:08

10

I think you should use a PathMatchingResourcePatternResolver.

answered Jul 29 '15 at 20:08

Luca Basso Ricci

17,829
2
47
69

score 5 · Answer 3 · answered Aug 27 '18 at 20:52

5

Alternatively @Value annotation can be used to let ResourceArrayPropertyEditor convert string to Resource[].

@Bean
@StepScope
public ItemReader<String> reader(
        @Value("file:/myfolder/*.csv") Resource[] resources
) {
    MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
    reader.setResources(resources);
    reader.setDelegate(new FlatFileItemReader<>(..));
    return reader;
}

answered Aug 27 '18 at 20:52

Ihar Zhavarankau

61
1
3

Is there any way to print the skip,commit,read,write count per file? – Ashish Jun 22 '19 at 11:37

Niraj Sonawane · Answer 4 · 2018-02-01T05:03:11.080

Read multiple files from Absolute file path:

 @Bean
      public MultiResourceItemReader<POJO> multiResourceItemReader() {
        MultiResourceItemReader<POJO> resourceItemReader = new MultiResourceItemReader<POJO>();
            ClassLoader cl = this.getClass().getClassLoader();
            ResourcePatternResolver resolver = new PathMatchingResourcePatternResolver(cl);

        Resource[] resources = resolver.getResources("file:" + filePath );
        resourceItemReader.setResources(resources);
        resourceItemReader.setDelegate(reader());
        return resourceItemReader;
      }

      @Bean
      public FlatFileItemReader<POJO> reader() {
        FlatFileItemReader<POJO> reader = new FlatFileItemReader<POJO>();
        reader.setLineMapper(new DefaultLineMapper() {{
          setLineTokenizer(new DelimitedLineTokenizer() {{
            setNames(new String[]{"id1", "POJO"});
          }});
          setFieldSetMapper(new BeanWrapperFieldSetMapper<POJO>() {{
            setTargetType(POJO.class);
          }});
        }});
        return reader;
      }

How to read all files in a folder with spring-batch and MultiResourceItemReader?

4 Answers4

Linked