Background
We have historically had trouble with deploying during operational hours (our setup: we have two EC2 instances behind a load balancer, which each instance having it's own beanstalkd queue. Each instance is configured using forge, and we autodeploy both using Envoyer.
Our Envoyer has a post deployment script that is supposed to automatically restart the queues after each deploy so that the jobs inside the queues can update to pick up on the latest code changes:
cd {{release}}
echo "" | sudo -S service php7.1-fpm reload
if [ -f artisan ]
then
php artisan config:cache
php artisan queue:restart
fi
The problem is that in random times weirdness happens in the system when we deploy during our operation hours. Ie people using our mobile apps would suddenly stop receiving push notifications (which are Laravel jobs processed by our queues) and other weirdness.
The work around was to restart the queues manually using the Forge queue UI panel. But this fix usually took some time to work, ie sometimes things started going back to normal 40 minutes after we manually restart the queues!
Analysis
I ran this experiment where I 1. ran an Envoyer deploy which executed the queue restart script, and observed the logs 2. manually restarted queues on forge, then observed the logs
Logs before Envoyer deploy:
forge 11556 0.0 0.0 4504 700 ? S 05:43 0:00 sh -c '/usr/bin/php7.1' 'artisan' queue:work 'beanstalkd' --once --queue='low' --delay=0 --memory=128 --sleep=9 --tries=3 --env='production'
forge 11557 3.0 1.2 384800 49476 ? S 05:43 0:00 /usr/bin/php7.1 artisan queue:work beanstalkd --once --queue=low --delay=0 --memory=128 --sleep=9 --tries=3 --env=production
forge 11570 0.0 0.0 4504 704 ? S 05:43 0:00 sh -c '/usr/bin/php7.1' 'artisan' queue:work 'beanstalkd' --once --queue='default' --delay=0 --memory=128 --sleep=6 --tries=3 --env='production'
forge 11571 3.6 1.2 384800 49496 ? S 05:43 0:00 /usr/bin/php7.1 artisan queue:work beanstalkd --once --queue=default --delay=0 --memory=128 --sleep=6 --tries=3 --env=production
forge 11575 0.0 0.0 4504 848 ? S 05:43 0:00 sh -c '/usr/bin/php7.1' 'artisan' queue:work 'beanstalkd' --once --queue='high' --delay=0 --memory=128 --sleep=2 --tries=3 --env='production'
forge 11576 5.5 1.2 384800 49764 ? S 05:43 0:00 /usr/bin/php7.1 artisan queue:work beanstalkd --once --queue=high --delay=0 --memory=128 --sleep=2 --tries=3 --env=production
forge 11580 0.0 0.0 4504 848 ? S 05:43 0:00 sh -c '/usr/bin/php7.1' 'artisan' queue:work 'beanstalkd' --once --queue='marketing' --delay=0 --memory=128 --sleep=3 --tries=3 --env='production'
forge 11581 12.0 1.2 384800 49448 ? S 05:43 0:00 /usr/bin/php7.1 artisan queue:work beanstalkd --once --queue=marketing --delay=0 --memory=128 --sleep=3 --tries=3 --env=production
forge 11586 0.0 0.0 12920 944 pts/1 S+ 05:43 0:00 grep --color=auto artisan
forge 20447 0.0 1.1 374208 46364 ? S Mar15 0:36 php /home/forge/default/envoyer/current/artisan queue:listen beanstalkd --sleep=9 --quiet --tries=3 --queue=low
forge 20633 0.1 1.1 374208 46604 ? S Mar15 1:52 php /home/forge/default/envoyer/current/artisan queue:listen beanstalkd --sleep=2 --quiet --tries=3 --queue=high
forge 20667 0.0 1.1 374208 46468 ? S Mar15 1:00 php /home/forge/default/envoyer/current/artisan queue:listen beanstalkd --sleep=3 --quiet --timeout=60 --tries=3 --queue=marketing
forge 20735 0.0 1.1 374208 46280 ? S Mar15 0:44 php /home/forge/default/envoyer/current/artisan queue:listen beanstalkd --sleep=6 --quiet --tries=3 --queue=default
Logs after Envoyer deploy, before manual queue restart:
forge 14367 0.0 0.0 4504 696 ? S 05:49 0:00 sh -c '/usr/bin/php7.1' 'artisan' queue:work 'beanstalkd' --once --queue='default' --delay=0 --memory=128 --sleep=6 --tries=3 --env='production'
forge 14368 2.0 1.2 384800 49900 ? S 05:49 0:00 /usr/bin/php7.1 artisan queue:work beanstalkd --once --queue=default --delay=0 --memory=128 --sleep=6 --tries=3 --env=production
forge 14387 0.0 0.0 4504 784 ? S 05:49 0:00 sh -c '/usr/bin/php7.1' 'artisan' queue:work 'beanstalkd' --once --queue='marketing' --delay=0 --memory=128 --sleep=3 --tries=3 --env='production'
forge 14388 5.5 1.2 384800 49528 ? S 05:49 0:00 /usr/bin/php7.1 artisan queue:work beanstalkd --once --queue=marketing --delay=0 --memory=128 --sleep=3 --tries=3 --env=production
forge 14392 0.0 0.0 4504 692 ? S 05:49 0:00 sh -c '/usr/bin/php7.1' 'artisan' queue:work 'beanstalkd' --once --queue='low' --delay=0 --memory=128 --sleep=9 --tries=3 --env='production'
forge 14393 11.0 1.2 384800 49680 ? S 05:49 0:00 /usr/bin/php7.1 artisan queue:work beanstalkd --once --queue=low --delay=0 --memory=128 --sleep=9 --tries=3 --env=production
forge 14397 0.0 0.0 4504 696 ? S 05:49 0:00 sh -c '/usr/bin/php7.1' 'artisan' queue:work 'beanstalkd' --once --queue='high' --delay=0 --memory=128 --sleep=2 --tries=3 --env='production'
forge 14398 12.0 1.2 384800 49504 ? S 05:49 0:00 /usr/bin/php7.1 artisan queue:work beanstalkd --once --queue=high --delay=0 --memory=128 --sleep=2 --tries=3 --env=production
forge 14403 0.0 0.0 12920 964 pts/1 S+ 05:49 0:00 grep --color=auto artisan
forge 20447 0.0 1.1 374208 46364 ? S Mar15 0:36 php /home/forge/default/envoyer/current/artisan queue:listen beanstalkd --sleep=9 --quiet --tries=3 --queue=low
forge 20633 0.1 1.1 374208 46604 ? S Mar15 1:52 php /home/forge/default/envoyer/current/artisan queue:listen beanstalkd --sleep=2 --quiet --tries=3 --queue=high
forge 20667 0.0 1.1 374208 46468 ? S Mar15 1:00 php /home/forge/default/envoyer/current/artisan queue:listen beanstalkd --sleep=3 --quiet --timeout=60 --tries=3 --queue=marketing
forge 20735 0.0 1.1 374208 46280 ? S Mar15 0:44 php /home/forge/default/envoyer/current/artisan queue:listen beanstalkd --sleep=6 --quiet --tries=3 --queue=default
Logs after Forge manual restart
forge 15012 0.6 1.1 374208 46652 ? S 05:50 0:00 php /home/forge/default/envoyer/current/artisan queue:listen beanstalkd --sleep=3 --quiet --timeout=60 --tries=3 --queue=marketing
forge 15097 0.7 1.1 374208 46368 ? S 05:50 0:00 php /home/forge/default/envoyer/current/artisan queue:listen beanstalkd --sleep=6 --quiet --tries=3 --queue=default
forge 15144 0.7 1.1 374208 46484 ? S 05:50 0:00 php /home/forge/default/envoyer/current/artisan queue:listen beanstalkd --sleep=2 --quiet --tries=3 --queue=high
forge 15269 1.1 1.1 374208 46452 ? S 05:50 0:00 php /home/forge/default/envoyer/current/artisan queue:listen beanstalkd --sleep=9 --quiet --tries=3 --queue=low
forge 15319 0.0 0.0 4504 744 ? S 05:50 0:00 sh -c '/usr/bin/php7.1' 'artisan' queue:work 'beanstalkd' --once --queue='default' --delay=0 --memory=128 --sleep=6 --tries=3 --env='production'
forge 15320 4.0 1.2 384800 49716 ? S 05:50 0:00 /usr/bin/php7.1 artisan queue:work beanstalkd --once --queue=default --delay=0 --memory=128 --sleep=6 --tries=3 --env=production
forge 15324 0.0 0.0 4504 776 ? S 05:50 0:00 sh -c '/usr/bin/php7.1' 'artisan' queue:work 'beanstalkd' --once --queue='high' --delay=0 --memory=128 --sleep=2 --tries=3 --env='production'
forge 15325 12.0 1.2 384800 49496 ? S 05:50 0:00 /usr/bin/php7.1 artisan queue:work beanstalkd --once --queue=high --delay=0 --memory=128 --sleep=2 --tries=3 --env=production
forge 15326 0.0 0.0 4504 700 ? S 05:50 0:00 sh -c '/usr/bin/php7.1' 'artisan' queue:work 'beanstalkd' --once --queue='low' --delay=0 --memory=128 --sleep=9 --tries=3 --env='production'
forge 15327 12.0 1.2 384800 49716 ? S 05:50 0:00 /usr/bin/php7.1 artisan queue:work beanstalkd --once --queue=low --delay=0 --memory=128 --sleep=9 --tries=3 --env=production
forge 15334 0.0 0.0 4504 692 ? S 05:50 0:00 sh -c '/usr/bin/php7.1' 'artisan' queue:work 'beanstalkd' --once --queue='marketing' --delay=0 --memory=128 --sleep=3 --tries=3 --env='production'
forge 15335 0.0 1.2 384800 49864 ? S 05:50 0:00 /usr/bin/php7.1 artisan queue:work beanstalkd --once --queue=marketing --delay=0 --memory=128 --sleep=3 --tries=3 --env=production
forge 15340 0.0 0.0 12920 1016 pts/1 S+ 05:50 0:00 grep --color=auto artisan
Conclusion: all the queue:work commands restart by the Envoyer php artisan queue:restart command. However, the artisan queue:listen commands are only restarted when we restart supervisor (ie via Forge ui manually).
Question
Why is Forge still using queue:listen although it's deprecated? How does the above explain how the auto-queue (queue:work) restart script isn't good enough, but restarting the supervisor/queue:listen does the job?
