curl_multi pause after a few executions

Question

My problem is I want to access json data from an API webpage, and there are only 10 calls per second allowed. After 10 calls added to the handler I exec them and then every result gets decoded, the data processed and the curl session will be removed from the handler.

In average it takes for one call about ~600 ms (results of 11600 calls). How to stop calling new ones after I called 10? I tried to use usleep (and it works) but it slows down everything really heavy. Somtimes about 20 seconds~ is this the only way to delay curl calls or how can I change this code to be more efficient?

$i = 0;
$run = NULL;
do {
  curl_multi_add_handle($handler, $curls[$i++]);
  while ($i < $curl_count && $i % 10 != 0) {
    curl_multi_add_handle($handler, $curls[$i++]);
  }
  curl_multi_exec($handler, $running);
  do {
    if (curl_multi_select($handler) != -1) {
      curl_multi_exec($handler, $running);
    }
  } while ($running > 0);

  while ($msg = curl_multi_info_read($handler)) {
    $exec_time[] = curl_getinfo($msg['handle'], CURLINFO_TOTAL_TIME);
    $response = curl_multi_getcontent($msg['handle']);
    $results = json_decode($response);
    processData() // function to process data
    curl_multi_remove_handle($handler, $msg['handle']);
  }
  $waiting_time = 1 - max($exec_time);
  if ($waiting_time > 0) {
    usleep(1000000 * $waiting_time);
  }
  $exec_time = NULL;
  usleep(100 * 1000);
} while ($i < count($curls));
curl_multi_close($handler);

EDIT Updated code to solve errors and improving it. See edits to see what has changed.

Mike Brant · Accepted Answer · 2014-08-25T14:28:33.430

0

You probably need to set up a timer and only sleep for the amount of time needed to get to 1 second interval, not a hard-coded 1 second time. So something like:

// start loop here

$start_time = microtime();

// run your 10 cURL requests here

$end_time = microtime();
$udelay = 1000000 * (1 - ($end_time - $start_time));
if ($udelay > 0) {
    usleep($delay);
}

// end loop here

By the way, that seems to be a fairly rudimentary usage of curl_multi. If you like you can take a look at this lightweight curl-based REST client I wrote. It would make this more robust for you with Exception handling and such.

https://github.com/mikecbrant/php-rest-client

edited Aug 25 '14 at 14:28

answered Aug 25 '14 at 14:18

Mike Brant

70,514
10
99
103

I tried your approach to measure the time it takes for the calls. But mostly the udelay time is about >=0.9 seconds, sadly slowing down the code alot (but working in the end). Anyhow I would need the curl call times directly, but there is no return value I could evaluate on. – pfu Aug 26 '14 at 11:10
@pfu Well you likely are going to be stuck with having long wait times if the curl calls only take up a fraction of the 1 second interval. At the end of the day, if you can only make 10 calls/second and you need to make 100 calls, that is going to take you 10 seconds. – Mike Brant Aug 26 '14 at 13:32
I just use now your approach but with the time curl gives me. It works now even better. :) Thanks for you advice. – pfu Aug 26 '14 at 14:23

curl_multi pause after a few executions

1 Answers1