Guzzle Pool : Wait for Requests

Question

Is it possible to make the Guzzle pool wait for requests?

Right now I can add requests to the pool dynamically, but as soon as the pool is empty, guzzle will stop (obviously).

This is a problem when I'm doing 10 or so pages concurrently, because my requests array will be empty until the resulting HTML pages have been processed and new links added.

This is my generator:

$generator = function () {
  while ($request = array_shift($this->requests)) {
    if (isset($request['page'])) {
      $key = 'page_' . $request['page'];
    } else {
      $key = 'listing_' . $request['listing'];
    }

    yield $key => new Request('GET', $request['url']);                                          
  }
  echo "Exiting...\n";
  flush();
};

And my pool:

$pool = new Pool($this->client, $generator(), [
  'concurrency' => function() {
    return max(1, min(count($this->requests), 2));
  },
  'fulfilled' => function ($response, $index) {
      // new requests may be added to the $this->requests array here
  }
  //...
]);

$promise = $pool->promise();
$promise->wait();

Edited code after answer by @Alexey Shockov:

$generator = function() use ($headers) {
  while ($request = array_shift($this->requests)) {
    echo 'Requesting ' . $request['id'] . ': ' . $request['url'] . "\r\n";

    $r = new Request('GET', $request['url'], $headers);

    yield 'id_' . $request['id'] => $this->client->sendAsync($r)->then(function($response, $index) {
      echo 'In promise fulfillment ' . $index . "\r\n";
    }, function($reason, $index) {
      echo 'in rejected: ' . $index . "\r\n";
    });
  }
};

$promise = \GuzzleHttp\Promise\each_limit($generator(), 10, function() {
  echo 'fullfilled' . "\r\n";
  flush();
}, function($err) {
  echo 'rejected' . "\r\n";
  echo $err->getMessage();
  flush();
});
$promise->wait();

Answer 1

Unfortunately, you cannot do that with a generator, only with a custom iterator.

I prepared a gist with the full example, but the main idea is just to create an Iterator that will change its state in both ways (it can became valid again after the end).

An example with ArrayIterator in psysh:

>>> $a = new ArrayIterator([1, 2])
=> ArrayIterator {#186
     +0: 1,
     +1: 2,
   }
>>> $a->current()
=> 1
>>> $a->next()
=> null
>>> $a->current()
=> 2
>>> $a->next()
=> null
>>> $a->valid()
=> false
>>> $a[] = 2
=> 2
>>> $a->valid()
=> true
>>> $a->current()
=> 2

With this idea in mind we can pass such dynamic iterator to Guzzle and leave it do the work:

// MapIterator mainly needed for readability.
$generator = new MapIterator(
    // Initial data. This object will be always passed as the second parameter to the callback below
    new \ArrayIterator(['http://google.com']),
    function ($request, $array) use ($httpClient, $next) {
        return $httpClient->requestAsync('GET', $request)
            ->then(function (Response $response) use ($request, $array, $next) {
                // The status code for example.
                echo $request . ': ' . $response->getStatusCode() . PHP_EOL;
                // New requests.
                $array->append($next->shift());
                $array->append($next->shift());
            });
    }
);
// The "magic".
$generator = new ExpectingIterator($generator);
// And the concurrent runner.
$promise = \GuzzleHttp\Promise\each_limit($generator, 5);
$promise->wait();

As I said before, the full example is in the gist, with MapIterator and ExpectingIterator.

Answer 2

It seems from the question, that you are able to move aggregation callback directly to the query. In this case the pool will always wait for your processing code, so you will be able to add new requests at any point.

A generator can return either a request or a promise, and promises can be combined together in different ways.

$generator = function () {
    while ($request = array_shift($this->requests)) {
        if (isset($request['page'])) {
            $key = 'page_' . $request['page'];
        } else {
            $key = 'listing_' . $request['listing'];
        }

        yield $this->client->sendAsync('GET', $request['url'])
            ->then(function (Response $response) use ($key) {
            /*
             * The fullfillment callback is now connected to the query, so the 
             * pool will wait for it.
             * 
             * $key is also available, because it's just a closure, so 
             * no $index needed as an argument.
             */
        });
    }
    echo "Exiting...\n";
    flush();
};

$promise = \GuzzleHttp\Promise\each_limit($generator(), [
    'concurrency' => function () {
        return max(1, min(count($this->requests), 2));
    },
    //...
]);

$promise->wait();

Answer 3

As I said before, the full example is in the gist, with MapIterator and ExpectingIterator

Iterators dont become valid again on php < 7, your example with arrayIterator and sample with MapIterator all stop after initial pool is exhausted...

On other hand it all works on earlier versions of php provided you use ->append method on iterator instead of [] push.

Answer 4

The answer is yes you can. You just need more generators. And to separate your request parse and queuing logic into asynchronous design. Rather than using an array for the requests your pool is going to issue and wait for it needs to be itself a generator that yields new requests from your initial list and requests added from parsed responses until all requests are sent parsed and resulting request are sent and parsed (recurring) or a stop condition is encountered.

Answer 5

If you can use postAsync/getAsync or so, you can use the below skeleton,

function postInBulk($inputs)
{
    $client = new Client([
        'base_uri' => 'https://a.b.com'
    ]);
    $headers = [
        'Authorization' => 'Bearer token_from_directus_user'
    ];

    $requests = function ($a) use ($client, $headers) {
        for ($i = 0; $i < count($a); $i++) {
            yield function() use ($client, $headers) {
                return $client->postAsync('https://a.com/project/items/collection', [
                    'headers' => $headers,
                    'json' => [
                        "snippet" => "snippet",
                        "rank" => "1",
                        "status" => "published"
                    ]        
                ]);
            };
        }
        
    };

    $pool = new Pool($client, $requests($inputs),[
        'concurrency' => 5,
        'fulfilled' => function (Response $response, $index) {
            // this is delivered each successful response
        },
        'rejected' => function (RequestException $reason, $index) {
            // this is delivered each failed request
        },
    ]);

    $pool->promise()->wait();
}