5

I am trying to figure out how to best setup the following scenario:

  1. Multiple A-type jobs are added into the queue
  2. When all A-type jobs are completed, a B or C-type job will be needed (one per A-type job)
  3. When all A, B and C type jobs are completed, a final D-type job will be needed

So basically we have some dependencies on jobs in the queue such that we don't want to start running jobs that require other jobs to be completed. Is there a guideline for setting up such a system? Should A-type jobs add B or C type jobs after their work is completed? Should all jobs be added up front and somehow tell the workers not to pull them until they are ready?

There are pros and cons of both approaches if I have to manually manage this dependency but I am curious if there is a different pattern I could use instead that might accomplish the same thing but in an easier way.

methodin
  • 6,717
  • 1
  • 25
  • 27

1 Answers1

5

I'm wondering if the routing slip pattern could inspire you here, ie send in the original A messages a "slip" that defines what the next messages (B, C) should be (what rules to apply to figure it out at runtime).

I would also add in the slip a unique correlation ID and the correlation group size (in that case: the total number of A messages).

For the final D job, I'd have the B/C process steps send a "D-ready" request message. When all the D-ready messages would have been collected (based on correlation ID/group size), I would trigger the final D job.

David Dossot
  • 33,403
  • 4
  • 38
  • 72
  • How would you recommend monitoring something like this assuming the job size of A's would be something like 3000 (thus 3000 B and Cs) and only one D? Standard table? – methodin Feb 08 '12 at 00:47
  • For global stats, you could use a different queue for each message type and use the queue stats. But knowing the state of each job would probably need an external data storage. – David Dossot Feb 08 '12 at 01:28