2

I have a scenario that I think is just outside the scope of spring-batch, but would like to explain my case and see if anyone has any ideas. I have a batch workflow like this:

  1. Load flat file into database
  2. Send message to another system to start doing its work
  3. Wait for other system to finish work (let's say that I can poll a database table to wait for a status flag to become a certain value)
  4. once other system is finished, then move to the next step in my job which reads some database records and writes a file

I've thought of these two approaches: 1. Split the above workflow into two spring batch jobs: one for steps 1-2 and another for step 4. Have some other process do the database polling to trigger the second job. 2. Do the polling inside of a tasklet that sleeps and then polls and returns a status indicating whether or not we need to poll again.

Option 1 is ok except that now I lose the ability to monitor the status of the overall "job" that I have. I'll have to create my own UOW progress monitoring that is tracking the individual spring-batch jobs.

Option 2 eats up a thread for each job. If I have hundreds of jobs that might be waiting at once I don't really want each polling individually and eating up a thread just to sleep.

I know the commercial workflow engine Flux has the concept of a workflow that hits a "wait" point that needs to be signalled externally and then it resumes. That would work well -- have some kind of external process (one thread) that is getting the set of jobs in a wait state, polling to see if any are ready to move on, and signalling the ones that are. Then spring batch picks up and continues on to the next step in the job.

Maybe I can approximate something by failing the job and having it restart? But then i can't tell the difference between a real failed job and just normal "wait" states in the spring batch admin web app.

Any clever ideas? Anyone know of any plans to add such functionality to spring-batch?

steve_ash
  • 1,184
  • 1
  • 10
  • 16
  • Why all the polling? Why not use messages for notifications instead? – Michael Minella Dec 16 '15 at 18:27
  • In this case I can't change the external system and it writes a record to the database when it's done. However, let's say I could use messages instead-- is there a way to accomplish having a step in the job "wait" until a message arrives and then go on -- or would I have to use two jobs then? (like Option2) – steve_ash Dec 17 '15 at 21:43
  • 1
    Yes. That's how partitioned steps work. They wait for all of the slaves to complete before moving on. We support that in partitioning via either messages or polling the job repository but messaging is definitely a better option IMHO. – Michael Minella Dec 17 '15 at 22:31

0 Answers0