dynamically load-balanced task queue

Question

Set up
Front-end machine receives web requests and puts 3 different kinds of tasks into Gearman server.
3 worker machines registered with that gearman server, each can perform any of those tasks.

Different tasks consume different resources.
Each task is different in terms of cpu/ram/io consumption.
The amount of tasks in queue is much greater then the number of workers, meaning each working machine registered multiple worker instances with gearman.

How to dynamically load balance it
The problem is that one worker machine can get all the cpu extensive tasks, another one would get the i/o extensive task, while ideally a worker can perform much better doing all this different kind of task in parallel.

Perhaps gearman isn't the right tool here?

I know rabbitmq has 'fair' balancing, where the next task goes to the machine that indicated it's free. But a machine can execute 3 tasks in parallel, each consuming different kind of resources. — Alex, Jan 21 '13 at 11:03

score 0 · Accepted Answer · answered Jan 21 '13 at 11:07

Possibly you could use RabbitMQ that has fair dispatching and
use 3 different queues, one for the cpu consuming task, one for the i/o and one for the network.

Thus every time a worker machine finishes any of this tasks, it would aknowledge it's done with it and get another task of the same resource-consuming type.

dynamically load-balanced task queue

1 Answers1