-1

Ok so I am a little stuck with this issue. I have a foreach loop (usually 50 results) that queries an API using Guzzle via Laravel Http and I am getting really inconsistent results.

I monitor the inserts in the database as they come in and sometimes the process seems slow and other times the process will fail with the following after x number of returned results.

cURL error 6: Could not resolve host: api.coingecko.com

The following is the actual code im using to fetch the results.

foreach ($json_result as $account) {

                    var_dump($account['name']);

                    $name = $account['name'];
                    $coingecko_id = $account['id'];
                    $identifier = strtoupper($account['symbol']);

                    $response_2 = Http::get('https://api.coingecko.com/api/v3/coins/'.urlencode($coingecko_id).'?localization=false');

                    if($response_2->successful()){

                        $json_result_extra_details = $response_2->json();

                        if( isset($json_result_extra_details['description']['en']) ){
                            $description = $json_result_extra_details['description']['en'];
                        }

                        if( isset($json_result_extra_details['links']['twitter_screen_name']) ){
                            $twitter_screen_name = $json_result_extra_details['links']['twitter_screen_name'];
                        }
                        
                    }else {
                        // Throw an exception if a client or server error occurred...
                        $response_2->throw();
                    }

                    $crypto_account = CryptoAccount::updateOrCreate(
                        [
                            'identifier' => $identifier
                        ],
                        [
                            'name' => $name,
                            'identifier' => $identifier,
                            'type' => "cryptocurrency",
                            'coingecko_id' => $coingecko_id,
                            'description' => $description,
                        ]);

                    //sleep(1);
    
                }

Now I know I am within the API rate limit of 100 calls a minute so I don't think that is the issue. I am wondering if this is a server/api issue which I don't really have any control over or if it related to my code and how Guzzle is implemented.

When I do single queries I don't seem to have a problem, the issue seems to be when it is inside the foreach loop.

Any advice would be great. Thanks

EDIT

Ok to update the question, I am now wondering if this is Guzzle/Laravel related. I changed the API to now point to the Twitter API and I am getting the same error after 80 synchronous requests.

ORStudios
  • 3,157
  • 9
  • 41
  • 69
  • Did you end up figuring out a solution to this? I'm getting very similar behaviour (after 80 successful requests with the API I'm calling) all of a sudden. Prior to yesterday this wasn't an issue at all... – shaneparsons May 11 '21 at 16:51
  • 1
    @shaneparsons no after getting the problem on two separate domain calls (coingecko & twitter) I assumed that it was more likely to be my end. So rather fetching all the account records and looping in one go I decided to use the scheduler and run every 2 minutes and fetch a limit of 40 records. This seems to do the trick and so far hasn't failed. – ORStudios May 12 '21 at 06:11
  • Also, was this a local / valet issue with you, or was is a staging / production one? In my case, this was only an issue locally w/ valet... I'm just trying to connect the dots. – shaneparsons May 13 '21 at 13:16
  • 1
    @shaneparsons For me this was local using Artisan. Never tried it on my server so I can't tell you if it was limited to the local version. – ORStudios May 14 '21 at 14:12

2 Answers2

-1

I think it's better to use Asynchronous Request directly with Guzzle.

$request = new \GuzzleHttp\Psr7\Request('GET', 'https://api.coingecko.com/api/v3/coins?localization=false');

for ($i=0; $i < 50 ; $i++) {
    $promise = $client->sendAsync($request)
        ->then(function ($response) {
            echo 'I completed! ' . $response->getBody();
        });
    $promise->wait();
}

more information on Async requests: Doc

Mahdi Jedari
  • 712
  • 1
  • 7
  • 16
-1

I have a similar problem as yours.

I doing the HTTP requests in the loop, and the first 80 requests are okay. But the 81st start throwing this "Could not resolve host" exception. It's very strange for me because the domain can be resolved perfectly fine on my machine.

Thus I start digging into the code. End up I found that Laravel's Http facades keep generate the new client. And I guess this eventually trigger the DNS resolver's rate limit?

So I have the workaround as following:

// not working
// as this way will cause Laravel keep getting a new HTTP client from guzzle.
foreach($rows as $row) {
    $response = Http::post();
}

// workaround
$client = new GuzzleHttp\Client();
foreach($rows as $row) {
    $response = $client->post();
    // don't forget use $response->getBody();
}

i believe it's because $client will cached the DNS resolve result, thus it will reduce the call to DNS resolver and not trigger the rate limit? I'm not sure whether it was right. BUT it's working for me.

crossRT
  • 616
  • 4
  • 12
  • 26