PHP How do I implement queue processing in php

Question

I want the data sent by my clients (via post) to be placed in a queue and a php script on my server first checks if the queue is empty. If the queue is not empty, then the script shall process all the data in the queue one by one.How do I do this?

Answer 1

This is something you could easily do with enqueue library. First, you can choose from a variety of transports, such as AMQP, STOMP, Redis, Amazon SQS, Filesystem and so on.

Secondly, That's super easy to use. Let's start from installation:

You have to install the enqueue/simple-client library and one of the transports. Assuming you choose the filesystem one, install enqueue/fs library. To summarize:

composer require enqueue/simple-client enqueue/fs 

Now let's see how you can send messages from your POST script:

<?php
// producer.php

use Enqueue\SimpleClient\SimpleClient;

include __DIR__.'/vendor/autoload.php';

$client = new SimpleClient('file://'); // the queue will store messages in tmp folder

$client->sendEvent('a_topic', 'aMessageData');

The consumption script:

<?php
// consumer.php

use Enqueue\SimpleClient\SimpleClient;
use Enqueue\Psr\PsrProcessor;
use Enqueue\Psr\PsrMessage;

include __DIR__.'/vendor/autoload.php';

$client = new SimpleClient('file://');

$client->bind('a_topic', 'a_processor_name', function(PsrMessage $psrMessage) {
   // processing logic here

   return PsrProcessor::ACK;
});

// this call is optional but it worth to mention it.
// it configures a broker, for example it can create queues and excanges on RabbitMQ side. 
$client->setupBroker();

$client->consume();

Run as many consumer.php processes as you by using supervisord or other process managers, on local computer you can run it without any extra libs or packages.

That's a basic example and enqueue has a lot of other features that might come in handy. If you are interested, check the enqueue documentation out.

Answer 2

You could use something like Zero MQ

See Example by Rasmus Lerdorf.

You could also consider using Gearman to distribute the load.

Answer 3

Take a look at this.

It uses memcached for persistence.

Answer 4

Problem with cronjob approach is, cronjob could at the most set to 1 minute interval, so there is a delay of 1 minute in the job execution, if that's acceptable then that's fine otherwise should use queues with polling script.

Answer 5

Here is another great tutorial for this:

http://squirrelshaterobots.com/programming/php/building-a-queue-server-in-php-part-1-understanding-the-project/

The other solution is using Gearman which they seem to have incorporated into PHP (it wasn't the last time I played with it): http://php.net/manual/en/book.gearman.php

Answer 6

Since relational DB's (Ex: MySQL) are so flexible, and well understood by web developers, they are used for many type of job queues. Many PHP applications use this solution as a fallback when object caching is unconfigured. It's the method of last resort because it's a very expensive way to implement a queue.

If you must use MySQL as your queue, one of the Percona engineers wrote this blog entry on managing the potential pain points.

If you want the most scalable implementation, I would highly recommend ZeroMQ. However it's not a default, or particularly common, PHP extension. So for a project where you would not be controlling the web server stack: Use APC Objects, Memcache or Memcached, and then fallback to a MySQL cache table.