Using Guzzle to perform batch request

Question

i'm working on a project where i need to perform 2000 asynchronous requests using Guzzle to an endpoint and each time i need to change the ID in the url param.

the endpoint looks like this: http://example.com/api/book?id=X

I tried to use a for loop to do that the only issue is that it's not asynchronous. what's the more efficient way to do such task?

public function fetchBooks () {
    $client = new Client();
    $responses = [];
    for ($i=1; $i < 2000; $i++) { 
        $response = $client->post('https://example.com/api/book?id=' . $i);
        array_push($responses, json_decode($response->getBody(), true));
    }
    return $responses;
}

Answer 1

In your case, I would first check, if the rest api exposes an endpoint, that makes it possible to send the data for all entries at once. In that case, you would need only a single call, or maybe a couple, if there is a limit on the amount of allowed entries to be attached.

If that is not the case, then there is no other way, then calling the rest endpoint 2000 times. You said it already. Your call is synchronous, which is inefficient, as it waits for each request to finish, before it starts the next.

To avoid that, use the async functions instead $client->postAsync. This will make it possible, to run the requests concurrently. Each request returns a promise. Basically, you want to wait for all promises to resolve and continue with the program execution. For that, store all promises in an array and use Promise\settle($promises)->wait(); or Promise\unwrap($promises);, if you want execution to stop, in case of an error.

The code could look like this:

<?php
use GuzzleHttp\Client;
use GuzzleHttp\Promise;

public function fetchBooks () {
    $client = new Client();
    $promises = [];
    for ($i=1; $i < 2000; $i++) { 
        $promise = $client->postAsync('https://example.com/api/book?id=' . $i);
        array_push($promises, $promise);
    }
    
    $results = Promise\settle($promises)->wait();
 
    $responses = [];

    foreach ($results as $result) {
      array_push($responses, json_decode($result->getBody(), true));
   }

   return $responses;
}


?>

Answer 2

@wasserholz answer is correct still using 2 array_push(takes more memory thus making api slower, so instead use []), & one array_push can be reduced.

use GuzzleHttp\Psr7\Response;

$responses = []; 
$client = new \GuzzleHttp\Client([
            'base_uri' => 'https://example.com'
        ]);
                
        $requests = function () use ($client) {
            for ($i = 0; $i < 2000; $i++) {
                yield function() use ($client,$i) {
                    return $client->postAsync('/api/book?id=' . $i);
                };
            }
            
        };
    
        $pool = new \GuzzleHttp\Pool($client, $requests(),[
            'concurrency' => 5,
            'fulfilled' => function (Response $response, $index) use (&$responses) {
                // if ($response->getStatusCode() == 200) {
                      $responses[] = json_decode($response->getBody(), true);
                // }
                print_r($responses); // this will have all the responses 
            },
            'rejected' => function (\GuzzleHttp\Exception\RequestException $reason, $index) {
                // dd($reason); //you can store it in laravel logs
            },
        ]);
    
        $pool->promise()->wait();
        return response()->json($responses);