Correct scope for multi threaded batch jobs in spring

Question

I believe I've got a scoping issue here.

Project explanation:

The goal is to process any incoming file (on disk), including meta data (which is stored in an SQL database). For this I have two tasklets (FileReservation and FileProcessorTask) which are the steps in the overarching "worker" jobs. They wait for an event to start their work. There are several threads dealing with jobs for concurrency. The FileReservation tasklet sends the fileId to FileProcessorTask using the job context.

A separate job (which runs indefinitely) checks for new file meta data records in the database and upon discovering new records "wakes up" the FileReservationTask tasklets using a published event.

With the current configuration the second step in a job can receive a null message when the FileReservation tasklets are awoken.

If you uncomment the code in BatchConfiguration you'll see that it works when we have separate instances of the beans.

Any pointers are greatly appreciated.

Thanks!

score 0 · Accepted Answer · answered Apr 10 '20 at 22:15

0

Polling a folder for new files is not suitable for a batch job. So using a Spring Batch job (filePollingJob) is not a good idea IMO.

Any pointers are greatly appreciated.

Polling a folder for new files and running a job for each incoming file is a common use case, which can be implemented using a java.nio.file.WatchService or a FileInboundChannelAdapter from Spring integration. See How do I kickoff a batch job when input file arrives? for more details.

answered Apr 10 '20 at 22:15

Mahmoud Ben Hassine

28,519
3
32
50

As an alternative to spring integration You can use 'Apache Camel' file endpoint to pick a file and then spring-batch to process it. – Sucheth Shivakumar Mar 13 '21 at 17:50

Correct scope for multi threaded batch jobs in spring

1 Answers1