0

I installed Gearman in my VPS server, after read documentation of Gearman I created two scripts to handle run 2,000 tasks in parallel.

This Worker script

<?php
$worker= new GearmanWorker();
$worker->addServer('localhost');
$worker->addFunction("fetchUrl", "my_reverse_function");
while ($worker->work());

function my_reverse_function($job)
{

 $curl_connection = curl_init($job->workload());
 curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
 curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
 curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);


 $data = json_decode(curl_exec($curl_connection), true);
 curl_close($curl_connection);
  return $data;
}
?>

This is client script which has 2,000 tasks

<?php

$client = new GearmanClient();
$client->addServer('localhost');

# set a function to be called when the work is complete
$client->setCompleteCallback("reverse_complete");

# Add some tasks for a placeholder of where to put the results
$results = array();
$client->addTask("fetchUrl", "http://**.com/getaccount.php?username=** ", &$results, "t1");
$client->addTask("fetchUrl", "http://**.com/getaccount.php?username=** ", &$results, "t2");
$client->addTask("fetchUrl", "http://**.com/getaccount.php?username=** ", &$results, "t3");
$client->addTask("fetchUrl", "http://**.com/getaccount.php?username=** ", &$results, "t4");
$client->addTask("fetchUrl", "http://**.com/getaccount.php?username=** ", &$results, "t5");
$client->addTask("fetchUrl", "http://**.com/getaccount.php?username=** ", &$results, "t6");
$client->addTask("fetchUrl", "http://**.com/getaccount.php?username=** ", &$results, "t7");

.... 
....

// till t2000 taks

$client->runTasks();

# The results should now be filled in from the callbacks
foreach ($results as $id => $result)
   echo $id . ": " . $result['handle'] . ", " . $result['data'] . "\n";


function reverse_complete($task, $results)
{
   $results[$task->unique()] = array("handle"=>$task->jobHandle(), "data"=>$task->data());
}

?>

Also I installed Gearman monitor script, everything is okay but the question how could handle and run all tasks in less time because this one it takes about 15 mins.

Regards

SimpleojbC
  • 127
  • 2
  • 14
  • 1
    Buy a bigger system :) Seriously: Lets say your VPS has 2 (virtual) CPUs. At the end this means you can run exactly 2 processes at once. The benefit from parallel processing comes, when IO-operations slows a process down and free some CPU-time for another process. For example you download a webpage. In this time another process will take over. However, the download will not be faster itself ;) Sidenote: Have a look at `curl_multi_exec()` – KingCrunch Aug 27 '13 at 11:20
  • @KingCrunch bigger system with the same code ? I mean will it do the process in less time ? – SimpleojbC Aug 27 '13 at 11:27
  • Probably you should also consider a bigger/faster network connection. – KingCrunch Aug 27 '13 at 11:41

0 Answers0