2

I'm working on an existing custom eCommerce PHP application which is currently running a very resource intensive CRON every 15 minutes.

The gist of it is: Customers can set up complex filters for products they are interested in, and receive emails based on these filters. The CRON which runs every 15 minutes, checks for all new products that have been listed since it last ran, and compares them with each customers filters. If the products match a customers filters they are send an email via amazon SES.

Up until now, this method has been working ok, but as the number of active customers is rising very quickly the CRON is starting to make a noticable performance drop on the application every 15 minutes which lasts for a minute or two while it runs.

I have been toying with other ideas to help spread out the load on the server, such as performing the task each time a product is listed, so the server doesn't need to catch up on multiple products at a time.

What is usually the best practise when approaching something like this?

Adam
  • 21
  • 2
  • I did something similar, except that I had the system setup to always send out a batch on a certain time, so my Cron will only run every 3 hours. – xlordt Mar 23 '17 at 13:16
  • 1
    Queues (using e.g. rabbitmq) and spreading the load across multiple servers. – solarc Mar 23 '17 at 13:37

2 Answers2

0

My recommended approach is to use a rabbitmq queue where your cron will send messages. Then set up a couple of consumers(scripts that will wait at the other end of the queue) that will take the message one by one, compose email and send it to the customer. This way, you can scale the number of consumers to match the volume of emails needed to be sent. If queues are not something you're familiar with, take a look at the RabbitMQ tutorials : https://www.rabbitmq.com/tutorials/tutorial-one-php.html

Alex
  • 187
  • 4
0

The message queues fit perfectly well and you can easily make use of them with enqueue library. Just a few words on why should you choose it:

  • It supports a lot of transports from the simplest one (filesystem) to enterprise ones (RabbitMQ or Amazon SQS).
  • It comes with a very powerful bundle.
  • It has a top level abstraction which could be used with the greatest of ease.
  • There are a lot more which might come in handy.

Instead of defining cron tasks you need a supervisord (or any other process manager). It must be configured to run a consume command. More on this in the doc.

Whenever a message is published it is delivered to a consumer (by a broker) and is being processed.

I suggest using RabbitMQ broker.

Maksim Kotlyar
  • 3,821
  • 27
  • 31