31

I can perform single requests using Guzzle and I'm very pleased with Guzzle's performance so far however, I read in the Guzzle API something about MultiCurl and Batching.

Could someone explain to me how to make multiple requests at the same time? Async if possible. I don't know if that is what they mean with MultiCurl. Sync would also be not a problem. I just want to do multiple requests at the same time or very close (short space of time).

halfer
  • 19,824
  • 17
  • 99
  • 186
Martijn
  • 2,268
  • 3
  • 25
  • 51
  • There's a [demo of this](http://guzzlephp.org/http-client/client.html#sending-requests-in-parallel) in the docs. This is still a synchronous call from your perspective, but will be parallel internally - thus the total time for the call will just be the time for the single longest fetch. – halfer Oct 23 '13 at 03:34

3 Answers3

30

From the docs: http://guzzle3.readthedocs.org/http-client/client.html#sending-requests-in-parallel

For an easy to use solution that returns a hash of request objects mapping to a response or error, see http://guzzle3.readthedocs.org/batching/batching.html#batching

Short example:

<?php

$client->send(array(
    $client->get('http://www.example.com/foo'),
    $client->get('http://www.example.com/baz'),
    $client->get('http://www.example.com/bar')
));
Community
  • 1
  • 1
Michael Dowling
  • 5,098
  • 4
  • 32
  • 28
29

An update related to the new GuzzleHttp guzzlehttp/guzzle

Concurrent/parallel calls are now run through a few different methods including Promises.. Concurrent Requests

The old way of passing a array of RequestInterfaces will not work anymore.

See example here

    $newClient = new  \GuzzleHttp\Client(['base_uri' => $base]);
    foreach($documents->documents as $doc){

        $params = [
            'language' =>'eng',
            'text' => $doc->summary,
            'apikey' => $key
        ];

        $requestArr[$doc->reference] = $newClient->getAsync( '/1/api/sync/analyze/v1?' . http_build_query( $params) );
    }

    $time_start = microtime(true);
    $responses = \GuzzleHttp\Promise\unwrap($requestArr); //$newClient->send( $requestArr );
    $time_end = microtime(true);
    $this->get('logger')->error(' NewsPerf Dev: took ' . ($time_end - $time_start) );

Update: As suggested in comments and asked by @sankalp-tambe, you can also use a different approach to avoid that a set of concurrent request with a failure will not return all the responses.

While the options suggested with Pool is feasible i still prefer promises.

An example with promises is to use settle and and wait methods instead of unwrap.

The difference from the example above would be

$responses = \GuzzleHttp\Promise\settle($requestArr)->wait();

I have created a full example below for reference on how to handle the $responses too. 

require __DIR__ . '/vendor/autoload.php';
use GuzzleHttp\Client as GuzzleClient;
use GuzzleHttp\Promise as GuzzlePromise;

$client = new GuzzleClient(['timeout' => 12.0]); // see how i set a timeout
$requestPromises = [];
$sitesArray = SiteEntity->getAll(); // returns an array with objects that contain a domain

foreach ($sitesArray as $site) {
    $requestPromises[$site->getDomain()] = $client->getAsync('http://' . $site->getDomain());
}

$results = GuzzlePromise\settle($requestPromises)->wait();

foreach ($results as $domain => $result) {
    $site = $sitesArray[$domain];
    $this->logger->info('Crawler FetchHomePages: domain check ' . $domain);

    if ($result['state'] === 'fulfilled') {
        $response = $result['value'];
        if ($response->getStatusCode() == 200) {
            $site->setHtml($response->getBody());
        } else {
            $site->setHtml($response->getStatusCode());
        }
    } else if ($result['state'] === 'rejected') { 
        // notice that if call fails guzzle returns is as state rejected with a reason.

        $site->setHtml('ERR: ' . $result['reason']);
    } else {
        $site->setHtml('ERR: unknown exception ');
        $this->logger->err('Crawler FetchHomePages: unknown fetch fail domain: ' . $domain);
    }

    $this->entityManager->persist($site); // this is a call to Doctrines entity manager
}

This example code was originally posted here.

The Unknown Dev
  • 3,039
  • 4
  • 27
  • 39
Bizmate
  • 1,835
  • 1
  • 15
  • 19
  • While this works great for my use case of loading multiple image URLs' data at once, how can one handle a situation where one of the URLs being loaded throws a 404 error? When this happens, Guzzle freaks out and throws a guzzle exception. I cannot guarantee the availability of the URLs so I was hoping to just load multiple requests and use the ones that actually go through. – georaldc Jan 18 '16 at 17:23
  • 1
    Nevermind, I just re-did my code to use GuzzleHttp\Pool instead. Seems to work well too and gives me a bit more control. – georaldc Jan 18 '16 at 18:29
  • 3
    can you share your code for multiple urls with Pool. – Sankalp Tambe Jan 25 '16 at 07:17
  • @georaldc i didnt like the Pool approach that much. Sankalp you can see a full example alternative that will return all responses instead of an exception in my answer update. – Bizmate Jun 05 '16 at 09:31
  • @Bizmate thanks for this it worked for me. I am curious on the level of concurrency and how much more performant it's than the old asynchronous way? – awm Nov 15 '17 at 16:28
11

Guzzle 6.0 has made sending multiple async requests very easy.

There are multiple ways to do it.

You can create the async requests and add the resultant promises to a single array, and get the result using the settle() method like this:

$promise1 = $client->getAsync('http://www.example.com/foo1');
$promise2 = $client->getAsync('http://www.example.com/foo2');
$promises = [$promise1, $promise2];

$results = GuzzleHttp\Promise\settle($promises)->wait();

You can now loop through these results and fetch the response using GuzzleHttpPromiseall or GuzzleHttpPromiseeach. Refer to this article for further details.

In case if you have an indeterminate number of requests to be sent(say 5 here), you can use GuzzleHttp/Pool::batch(). Here is an example:

$client = new Client();

// Create the requests
$requests = function ($total) use($client) {
    for ($i = 1; $i <= $total; $i++) {
        yield new Request('GET', 'http://www.example.com/foo' . $i);
    }
};

// Use the Pool::batch()
$pool_batch = Pool::batch($client, $requests(5));
foreach ($pool_batch as $pool => $res) {

    if ($res instanceof RequestException) {
        // Do sth
        continue;
    }

    // Do sth
}
Annapurna
  • 529
  • 1
  • 6
  • 19