0

Is it possible to idle the Client when all workers are working?

I have the following case:

$client= new GearmanClient();
$client->addServer();

$data = array(thousands of entries);

foreach($data as $dt) {
    $client->doBackground('doFancy', $dt);
}

When I run this, it will make thousands of queues or even overload the memory.

Is it possible to wait for a worker to become free until the foreach loop continues assigning new jobs?

Aley
  • 8,540
  • 7
  • 43
  • 56
  • That does seem to go against the entire point of using a queue. How big are each of the tasks you are sending? Could you perhaps package them into larger chunks to reduce the amount of tasks you request? – Louis-Philippe Huberdeau Feb 20 '13 at 19:07
  • @Louis-PhilippeHuberdeau each task will run for aprox. 10 seconds, but when doing it like it is currently, I will push 300000 tasks at ones. – Aley Feb 20 '13 at 19:16
  • if the php script that handles the information can handle the size of the information, I have a hard time believing that Gearman daemon will not handle it. It's much more robust than a php script.. – Aurimas Feb 20 '13 at 22:35

1 Answers1

0

You can create counter for queue size and set callback for complete event. In this callback you have to decrements counter and check the counter and if it's value less than N - new bunch of tasks will added and counter increments.

Kirill Zorin
  • 239
  • 1
  • 5
  • If I get it right a background worker has to send back a callback to the client. Is this possible? May you write a small example (just pseudocode or so)? I would much appreciate it. – Aley Feb 24 '13 at 16:17
  • 1
    Worker code: `$worker->addFunction('myfunc', 'handler');` And in handler function: `function handler(GearmanJob $job) { $job->sendComplete('Any complete message'); }` Also in client: `function completeHandler(GearmanTask $task) { ... } $client->setCompleteCallback('completeHandler');` – Kirill Zorin Feb 24 '13 at 18:29
  • I'm not sure if this will work for background jobs. The client creates the background jobs and will not wait until the jobs are done. So when the worker excute "sendComplete" the client will be already done with execution. – Aley Feb 25 '13 at 11:12
  • Right, but you can create client that will be runned permanently. Like a workers. In one of my projects I use that way and It's works great. – Kirill Zorin Feb 25 '13 at 12:12
  • Oh can I keep the client alive with a while loop? May I ask you to make another small example (just for the client)? I would much appreciate it. – Aley Feb 26 '13 at 00:03
  • I'am use client class that extends a GearmanClient. In this class I have method _start_ with _while(true) {}_ loop, method _process_ for execution in loop and method _complete_ for complete callbacks. And client stating looks like this: `$client->start();` – Kirill Zorin Feb 26 '13 at 07:22
  • 1
    Also, for processes control I'am use the Supervisord – Kirill Zorin Feb 26 '13 at 07:24
  • Thanks Kirill! That should be enough to continue :) – Aley Feb 26 '13 at 13:14