I created a Beanstalk queue with Laravel Forge with the following settings:
90 seconds max per job
10 rest seconds when empty
3 maximum tries
production environment
I ran a job from my Laravel 5.2 app that did not process correctly (a payment profile was not created for a customer). However, the job was never added to my failed_jobs table in my database. Then, I ran a few more jobs after to see what might have happened, and those jobs did not run either. It seems like Beanstalk was stuck trying to process the first job and could not get over it to move to the next one. Still no jobs in the failed_jobs table in my database.
I checked on the queue in Forge and it said that the queue was still running. So I logged into telnet and checked out the Beanstalk queue. I did a peek-ready and saw the first job that did not process sitting right there. But it was not processing or failing due to the 3 maximum tries setting. It was just stuck.
After this, I figured I would try to restart my Beanstalk queue in Forge. As soon as the queue restarted, all of the jobs that had gotten stuck processed successfully.
I'm glad that I was able to alleviate that bottleneck, but I'm concerned because I don't want this happening to my app while it's live. What might have happened here? What can I do to prevent it, or how can I set up some kind of alert to let me know that the queue is stuck?
EDIT:
I just took a look at the settings for the queue in /etc/supervisor/conf.d. Everything looks normal, right?
[program:worker-49340]
command=php /home/forge/mydomain.com/current/artisan queue:listen beanstalkd --timeout=90 --sleep=10 --quiet --tries=3 --queue="default"
autostart=true
autorestart=true
user=forge
redirect_stderr=true
stdout_logfile=/home/forge/.forge/worker-49340.log
I also checked /home/forge/.forge/worker-49340.log to see if it had any info, but it was empty.
I can't currently reproduce the same situation even though I'm providing the exact same input that caused the queue to get stuck in the first place.
