3

I needed to know if laravel 5 queue management system is suitable for big projects (having about 100.000 users). I want to do something like messaging (not spam :) ) users at once each day.
Is redis good enough for this job (queuing)? Or it is better to use a lib that is specially for queuing (like beanstalkd)?

Ali Farhoudi
  • 5,350
  • 7
  • 26
  • 44

1 Answers1

7

To be fair and to try and post a reasonable answer to this question we should consider the following:

  • The number of subscribers
  • The content to be delivered
  • The system resources required to run simultaneous queues

100,000 subscribed emails would require storing 100,000 x [data] in RAM, so lets average out the email address length to 32 characters (bytes).

100,000 x 32 bytes = 3.2MB

Of course, Laravel's queue system serialises objects, so actual memory usage will probably be higher (Redis memory used for Laravel queue), but not enough to concern yourself with.

I've advised in the past that a seemingly successful setup for sending out subscribed email messages would run on the following:

  • 2GB RAM minimum
  • 2 processors / cores

The queue system Laravel runs is not too taxing on a server. As always, scale with requirements.

The software for such (using Laravel) would consist of the following:

  • Redis
  • Supervisor

Set up Redis as Laravel's queue driver. Remember to composer require predis/predis.

You will also need to create a migration for storing failed jobs. Laravel has one built in by default:

php artisan queue:failed-table

php artisan migrate

Once Supervisor is installed, create a conf file in /etc/supervisor/conf.d so that Supervisor can pick up on the configuration for your queue:

touch /etc/supervisor/conf.d/myprojectqueue.conf nano /etc/supervisor/conf.d/myprojectqueue.conf

In there, lay out a configuration to suit your environment. In the following demo set up, 4 queue runners will execute on your queue at once:

    [program:myprojectqueue]
    command=php /path/to/project/artisan queue:listen --tries=1
    directory=/path/to/project
    stdout_logfile=/path/to/project/storage/logs/supervisord.log
    redirect_stderr=true
    autostart=true
    autorestart=true
    numprocs = 4
    process_name = %(program_name)s%(process_num)s

Save the conf file. Start / Restart Supervisor.

For more information:

https://laravel.com/docs/master/queues

https://laravel.com/docs/master/queues#supervisor-configuration

https://laravel.com/docs/master/mail#queueing-mail

https://laravel.com/docs/master/scheduling

Justin Origin Broadband
  • 1,492
  • 1
  • 11
  • 14
  • Thank you @Justin. That was a complete and compelling answer. I tried laravel's queuing with redis. And I checked laravel codes, too. I figured out that laravel is managing queue by itself and redis is only a backed database. I think it makes the queue slower. Just wanted to know what about if i want to implement the queue management system separated from my main project. For example implementing with python (django) and celery? Or with cpp and beanstalk? – Ali Farhoudi Jun 15 '16 at 10:02
  • @ali farhoudi, could you resolve your problem? multiple queue slower than single queue with this solution? – DolDurma Jul 31 '18 at 03:35
  • 1
    @DolDurma, Yes, I finally used laravel with beanstalked queue driver. We have been using them for about tow years and we had no problem. Currently the queue is processing about 100,000 jobs in about 1 hour and half with only 7 workers. And every job takes about 2 seconds. We had to reduce workers because of SLA rules. With more amount of workers, the process was much faster (considering CPU and RAM are OK). – Ali Farhoudi Jul 31 '18 at 04:47
  • @AliFarhoudi so, can i get your mail address? your website is not online currently, Thanks in advace – DolDurma Jul 31 '18 at 06:40
  • @DolDurma Sure, af{dot}farhoudi{at-sign}gmail{dot}com – Ali Farhoudi Jul 31 '18 at 07:17