0

I have a problem with this function: 

function multi_activity($nodes,$headers){

$node_count = count($nodes);
$results=array();
$curl_arr = array();
$master = curl_multi_init();

for($i = 0; $i < $node_count; $i++)
{
    $url =$nodes[$i];
    $curl_arr[$i] = curl_init($url);
    curl_setopt($curl_arr[$i], CURLOPT_RETURNTRANSFER, true);
    curl_setopt($curl_arr[$i], CURLOPT_HEADER, false);
    curl_setopt($curl_arr[$i], CURLOPT_HTTPHEADER, $headers);
    curl_setopt($curl_arr[$i], CURLOPT_FOLLOWLOCATION, false);
    curl_setopt($curl_arr[$i], CURLOPT_SSL_VERIFYPEER, false);
    curl_multi_add_handle($master, $curl_arr[$i]);
}    
  do {       
    curl_multi_exec($master,$running);  
    } while($running > 0 );  


for($i = 0; $i < $node_count; $i++)
{

    $results[] = curl_multi_getcontent  ( $curl_arr[$i]  );
    curl_multi_remove_handle($master, $curl_arr[$i]);

}   
curl_multi_close($master);
return $results;
}

Actually, I'm calling this function 300 times and each time with a $nodes containing 30 different URLs. I don't actually understand How CUrl actually works But I have to wait for 10mn to get the job done and to print my JSON. Is there a way to improve it by using multi_curl or an other asynchronous PHP tool/API. Thanks in advance.

Sarah Aziziyan
  • 498
  • 9
  • 22
Random_Display_Name
  • 57
  • 1
  • 9
  • 1
    Scrapping 9000 URLs in 10 minutes is still 15 per second. That's pretty good! You'd have to see what is blocking you, if it's internet speed, CPU or memory... – Salketer Sep 13 '17 at 12:55
  • well... i was expecting less than a minute to be honest. How can i check that ? IS this a limit from libcurl or is it my loop whos causing this ? – Random_Display_Name Sep 13 '17 at 13:31
  • Less than a minute for 9000 URLs would be incredibly fast... The most limiting part is coming from connection negotiation and the peer servers response speed/size. – Salketer Sep 13 '17 at 13:49
  • 1
    if you call that function just once and add all 300 * 30 handles that will be huge speed up – Kazz Sep 13 '17 at 14:16
  • @Kazz You mean actually set up the entire array before calling the function ? – Random_Display_Name Sep 13 '17 at 14:39
  • yea like `for($j = 0; $j < $node_count * 300; $j++){ $i = $j % $node_count;` – Kazz Sep 13 '17 at 14:47
  • Just note that adding 300 * 30 = 9000 handles in one go will cause problems due to lack of file descriptors in most default installs... – Daniel Stenberg Sep 15 '17 at 06:26

0 Answers0