How to make multiple gearman worker to work on single job

Question

I am processing 1 lakh urls using perl gearman client and worker .

I need your help to run single job in multiple worker . (ie if i have 5 workers and 1 client i want all these 5 workers to do the job of one client ),currently I am running 20 clients and 30 workers but only 20 workers are running the job balance 10 workers are idle.

Thanks in advance

If you have a specific problem, post your code so we can help you. If not, we are not going to write your code. — Benjamin Schwalb, Jul 10 '13 at 08:16
two germans, working on a job together, I think I need to sleep, :D — aah134, Jul 10 '13 at 08:26

score 0 · Answer 1 · answered Jul 10 '13 at 08:20

0

Gearman worker grabs one job, and takes it as an execution unit. If you would like to run one job on multiple workers, you should probably divide your job into several sub-jobs.

answered Jul 10 '13 at 08:20

score 0 · Answer 2 · answered Jul 29 '13 at 08:20

0

You can create a manager which manages jobs and coordinates other workers.

answered Jul 29 '13 at 08:20

itechnolohiya

1

score 0 · Answer 3 · answered Dec 17 '13 at 11:25

0

The approach you need is called fan-out, but gearman cant do this. you have to use rabbitmq like message queue. it can send a same message to different workers with fanout exchanges.

answered Dec 17 '13 at 11:25

guy_fawkes

947
8
31

How to make multiple gearman worker to work on single job

3 Answers3