How to continuously read files from a folder with spring-batch?

Question

I'm using spring-batch to process multiple csv files from a folder using spring-batch. Similar as follows:

    @Bean
    public ItemReader<String> reader() {
        MultiResourceItemReader<String> reader = new MultiResourceItemReader<>();
        reader.setResources(new Resource[] {new FileSystemResource("/myfolder/*.csv")});
        reader.setDelegate(new FlatFileItemReader<>(..));
        return reader;
    }

The jobs is a single thread executor.

Question: whenever new files are added to the folder, how can I add them to the list and import them automatically using the job?

It's about 30.000 files per day in this folder. Can I tell spring-batch to automatically detect new files?

Or would I always have to restart the job as soon as it has finished, so that it then starts to import the files the have been added while the job ran?

score 3 · Accepted Answer · answered May 03 '17 at 07:25

A SpringBatch is a "static" thing. Once the job has started, it is defined what it will do and this cannot be changed (... without doing nasty hacking).

Therfore, you are not able to change the definition of your MultiResourceItemReader, once the job has started.

Hence, one execution of the jobs processes only the files that were present when the job was started and in order to process files that arrived later, another execution of the job needs to be triggered.

It's important to understand, that SpringBatch is not a job control system. SpringBatch is here to define the jobs and execute a job run. However, in order to trigger a job to be started, you need something else.

As was suggested, that could be a cron-job, spring-integration, springs Scheduler-Annotations, ... as suggested by Stimpson Cat.

Moreover, based on my own experience I would also recommend using a directory structure as proposed by Essex Boy

score 2 · Answer 2 · answered May 02 '17 at 14:04

You could solve this using a cron timer that runs your spring batch application periodically. Therefore you need to move the already processed files. Or you can use the java 7 file watcher Java Documentation

But with this setup you will have to handle the events yourself. For example when a new file arrives.

score 2 · Answer 3 · answered May 02 '17 at 14:09

A little off topic, but when you do this create directories for

1) INPUT

2) INPROGESS

3) DONE

4) ERROR

Then run your Java job from the CRON, as suggested by Stimpson Cat. The Jave job will look for any files in INPUT, move them into INPROGRESS, process them then more the to DONE or ERROR.

How to continuously read files from a folder with spring-batch?

3 Answers3