0

I am trying to implement a load balancer at the moment and have hit a bit of a speed bump. The situation is as follows (simplified),

  • I have a queue of requests queue_a which are processed by worker_a
  • There is a second queue of requests queue_b which are processed by worker_b
  • And I have a third queue of requests queue_c that can go to either of the workers

The reason for this kind of setup is that each worker has unique requests that only it can process, but there are also general requests that anyone can process.

I was going to implement this basically using 3 instances of the C5 IntervalHeap. Each worker would have access to its local queue + the shared queues that it is a part of (e.g., worker_a could see queue_a & queue_c).

The problem with this idea is that if there is a request in the local queue and a request in the shared queue(s) with the same priority, it's impossible to know which one should be processed first (the IntervalHeap is normally first-come-first-serve when this happens).

EDIT: I have discovered IntervalHeap appears to not be first-come-first-server with same priority requests!

I would like to minimise locking across the queues as it will be relatively high throughput and time sensitive, but the only way I can think of at the moment would involve a lot more complexity where the third queue is removed and shared requests are placed into both queue_a and queue_b. When the request is sucked up it would know it is a shared request and have to remove it from the other queues.

Hope that explains it clearly enough!

mike
  • 3,146
  • 5
  • 32
  • 46
  • 1
    Take a look at TPL dataflow, it provides generic solutions for your stated problem. You might have to drop the use of C5 in that case – Polity Jul 10 '12 at 05:55
  • Do you have any more information on this? e.g., how I could use TPL in this scenario? I have googled it a bit but it's not clear how I could use this in my situation – mike Jul 10 '12 at 08:10

2 Answers2

1

It seems that you'll simply end up pushing the bubble around - no matter how you arrange it, in the worst case you'll have three things of equal priority to execute by only two workers. What sort of tie breaking criteria could you apply beyond priority in order to choose which queue to pull the next task from?

Here are two ideas:

  1. Pick the queue at random. All priorities are equal so it shouldn't matter which one is chosen. On average in the worst case, all queues will be serviced at roughly the same rate.

  2. Minimize queue length by taking from the queue that has the largest number of elements. This might cause some starvation of other queues if one queue's fill rate is consistently higher than others.

HTH

Monroe Thomas
  • 4,962
  • 1
  • 17
  • 21
  • yes every solution seems to have limitations. I have thought of one where the shared queues insert a kind of token into the other queues to represent available work. When the worker sees these tokens in their local queue, they try to get the request from the shared queue. If it doesn't exist anymore, they simply continue on. Otherwise it is processed normally. This just introduces overhead if you have shared queues across a large number of other queues, as every queue needs to check every shared queue request. – mike Jul 10 '12 at 08:26
  • I disagree with this answer. There certainly is a priority difference between private and shared items. It all depends on the state of the different queues. See my answer. – Polity Jul 11 '12 at 02:51
  • @Polity The OP explicitly stated that the priority of items at the front of the queues were equal. Saying that the private queue is higher priority may be a perfectly valid way to solve the problem, but is not a value judgement supported by the original question - which essentially boils down to: I have two equal priority things to do, but only one worker - which should it choose? There needs to be some tie breaking mechanism, and amongst a plethora of choices, choosing the private queue is no more or less valid than choosing randomly or choosing the longest queue. – Monroe Thomas Jul 12 '12 at 05:27
1

Synchronizing your workers can share the same pool of resources as well as their private queue. Of there is 1 item available in the queue for worker 1 and 1 item available in the shared queue, it would be a shame if worker 1 picks up the item of the shared queue first since this will limit parallel runs. Rather you want worker 1 to pick up the private item first, this however leads to new caveats, one being where worker 1 and worker 2 are both busy handling private items and therefore older shared items will not be picked up.

Finding a solution that addresses these problems will be very difficult when also trying to keep the complexity down. A simple implementation is only to handle shared items when the private queue is empty. This does not tackle the part where priorities are not handled correctly on high load scenario's. (e.g. where the shared queue wont be handled since the private queues are always full). To balance this, you might want to handle the private queue first, only if the other workers private queue is empty. This is still not a perfect solution since this will still prefer private queue items over shared items. Addressing this problem again can be achieved by setting up multiple strategies but here comes even more complexity.

It all depends on your requirements.

Polity
  • 14,734
  • 2
  • 40
  • 40
  • I've currently implemented it so the shared queue is not a queue but more of a lookup list. When the shared queue receives work it puts the work in all of the underlying queues, with a special flag to indicate it is shared work. When a worker receives a workload with these special flags, it checks against the shared queue to ensure the work still needs to be done. If it's no longer valid, it just discards & grabs the next item in it's queue. If valid, it continues processing it. This results in a couple of extra checks happening, but maintains the functionality I was after. – mike Jul 12 '12 at 04:34