How to NOT return data from curl-multi and reduce CPU usage?

Question

I have a script in php that uses multi curl.

A general multicurl request consists of 50 individual curl handles. It takes some time for each request to return, so my script should just wait until processing is finished on the remote server.

Each of those 50 requests returns quite a lot of data (content) which I don't need. Therefor it would be more efficient to ignore the returned data. However I do need to know when processing on the remote server is finished, which is when data is returned.

The reason why I don't need to use the data, but do need to make the request is that the remote server puts data in a database, which I subsequently take from my own server. So I basically only need to make this request AND I need to know when the script on the remote server is finished.

My question: This request is consuming a lot of CPU how can I make it more efficient?

The code:

$nrofrequests=count($variable1);

//Build multi-curl for all to scrape sites at once:
for($i=0;$i<$nrofrequests;$i++){

$post='variable1='.$variable1[$i].'&variable2='.$variable2[$i];
$url='http://www.domain.com/'.$scriptnumber[$i];


$ch[$i] = curl_init($url);
curl_setopt ($ch[$i], CURLOPT_POST, 1);     
curl_setopt ($ch[$i], CURLOPT_POSTFIELDS, $post); 
curl_setopt($ch[$i], CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch[$i], CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch[$i], CURLOPT_TIMEOUT, 35);  
set_time_limit(35); 
}


// build the multi-curl handle, adding both $ch
$mh = curl_multi_init();
for ($i=0; $i<$nrofrequests; $i ++ ):
curl_multi_add_handle($mh, $ch[$i]);
endfor;


// execute all queries simultaneously, and continue when all are complete
$running = null;
do {curl_multi_exec($mh, $running);} 
while ($running);


for ($i=0; $i<$nrofrequests; $i ++ ):
curl_multi_remove_handle($mh, $ch[$i]);
endfor;

Perhaps helpful: http://curl.haxx.se/mail/lib-2009-02/0166.html. It's in C, but the principle should be the same in PHP. — jimw, May 10 '12 at 11:27

Zombaya · Answer 1 · 2012-05-10T21:24:21.373

2

If you don't set CURLOPT_RETURNTRANSFER to true and set CURLOPT_NOBODY to true, you only get a boolean as result whether or not it failed. But I doubt that will improve CPU-usage.
Sleeping while waiting for the curl to finish will however improve CPU-usage.

// During init
curl_setopt($ch[$i], CURLOPT_RETURNTRANSFER, false); 
curl_setopt($chi[$i], CURLOPT_NOBODY,true);

// execute all queries simultaneously, and continue when all are complete
$running = null;
do {
    curl_multi_select($mh,5);  // Wait max 5 seconds 'till at least one of the curls shows activity
    curl_multi_exec($mh, $running);
} 
while ($running );

edited May 10 '12 at 21:24

answered May 10 '12 at 11:54

Zombaya

2,230
23
30

this is highly inefficient and I really cannot recommend this way – Daniel Stenberg May 10 '12 at 17:33
@DanielStenberg, why? You still need to wait for all the curl-request to finish and while waiting it sleeps, so I don't really get why this is inefficient. If you care to explain why, I will edit or remove my answer if it really is. – Zombaya May 10 '12 at 17:56
because it sleeps unconditionally for a long time, which then basically will make this code always take about half a second longer than if you had used the curl_multi_select way properly – Daniel Stenberg May 10 '12 at 20:32
True, but if you want that you could lower the sleep value to avoid that. – Zombaya May 10 '12 at 20:57
Incorporated your suggestion. – Zombaya May 10 '12 at 21:01
and with that, my objection is hereby withdrawn =) Thanks! (you missed an underscore though in curl_multi_select) – Daniel Stenberg May 10 '12 at 21:18

score 0 · Answer 2 · answered May 10 '12 at 17:33

0

This has been answered many times already on stackoverflow. You MUST use curl_multi_select to avoid busy-looping.

answered May 10 '12 at 17:33

Daniel Stenberg

54,736
17
146
222

How to NOT return data from curl-multi and reduce CPU usage?

2 Answers2