0

I am trying to add data from 2 APIs to my database (using CraftCMS). I am querying 2 endpoints - one of which queries all posts (/postings) then another queries each post (/postings/). I am currently looping through each item in the postings API then querying the API:

foreach ($data AS $item) {
  $this->queryPostingItem($item->id);
}

I am using Guzzle to fetch the data in each method. But getting a curl timeout error. What options do I have to fetch all the posting data? I believe that there is a limit of 2 api calls per second. I have tried adding sleep(1); to the loop after every few loops but it isn't helping.

Bit stuck.

Peter Ayello Wright
  • 127
  • 1
  • 3
  • 13
  • ```foreach ($data AS $item) { while(!$this->queryPostingItem($item->id)){ /*probably hitting api rate limit, sleep a bit and try again*/sleep(3); } } ``` – hanshenrik Aug 23 '21 at 14:12
  • Thanks @hanshenrik I have tried - ```$newData = array_chunk($data, 2); foreach ($newData AS $data) { sleep(4); foreach ($data AS $key => $item) { $this->queryPostingItem($item->id); } ``` But still get a timeout error – Peter Ayello Wright Aug 23 '21 at 14:22
  • what timeout error? show us the CURLOPT_VERBOSE response, are you talking about a server-generated timeout error (which would imply api ratelimiting), or are you talking about a libcurl-generated timeout? (which would imply that you need to increase CURLOPT_TIMEOUT / CURLOPT_CONNECTTIMEOUT), or are you talking about a php-timeout? (which means you'll need to increase php's set_time_limit()) or are you talking about your own web server's limit, eg nginx/apache php gateway timeout? – hanshenrik Aug 23 '21 at 14:52
  • cURL error 28: Operation timed out after 2047 milliseconds with 0 bytes received – Peter Ayello Wright Aug 23 '21 at 15:02
  • "*I believe that there is a limit of 2 api calls per second*" - sounds like there are rate limits on this API, and they are documented? Sometimes there are per-second rate limits as well as total request limits, are you now hitting the latter? Maybe a question for their support. – Don't Panic Aug 23 '21 at 15:29
  • @PeterAyelloWright try setting CURLOPT_CONNECTTIMEOUT to 5 and run it again, what do you get? – hanshenrik Aug 23 '21 at 18:35
  • you can use concurrency it will send exactly 2 at a time then put a sleep in every concurrent loop of guzzle, would solve the issue – bhucho Aug 24 '21 at 07:54
  • Ok I did more digging and found - $response = $client->send($request, ['timeout' => 10,'headers' => [ $headers ]]); and set the timeout to 10 seconds. Was previously not set. This seems to load now or at least doesn't error out. – Peter Ayello Wright Aug 24 '21 at 08:18

0 Answers0