0

I have currently a tough time with implementing the following scenario in Java:

I have a single producer with multiple consumer problem - but not not every consumer can process every job:

There are two types of jobs:

  • xxl-jobs require a very large amount of memory
  • normal-job don't require that much memory

And of course two types of consumers:

  • xxl-consumer can process any type of job
  • normal-consumer can only process normal-jobs

Is there an easy way to put that scenario in code?

The obvious approach would be to use two queues (one for each job type) and filter the jobs. However this would mean that the xxl-consumer would have to query two queues for new jobs - which is AFAIK not possible in Java in an efficient way (the posted work-arounds here on SO don't work as I have the normal-consumers who only listen for one queue).

JMax
  • 1,134
  • 1
  • 11
  • 20
  • 3
    Logical way to solve it is to have a broker that queries for every job and deals jobs to consumers of appropriate type. If no appropriate consumer is available (because this is an xxl job, and all xxls already busy), then broker can either but job back on the input queue, or have an internal pressure queue for that. – M. Prokhorov Jul 05 '17 at 12:19
  • You did not mention if the jobs have a priority or a time window in which they must be executed. If so, then you must also consider some sort of a priority queue so that the jobs that must be executed first are executed first. – Jim Rogers Jul 05 '17 at 18:16
  • @Jim Rogers Luckily timing isn't a problem in my scenario. Each job takes minutes to hours and most of the jobs are created right when the system is started. – JMax Jul 06 '17 at 07:50

0 Answers0