Laravel systemd queue job stops, when using restart=on-failure

Question

We have a Laravel SystemD job for the queue, which for example sends out notification email.

Now we noticed, that the SystemD job is stopped when a email delivery did not work.

This is how our service definition looks like:

/etc/systemd/system # cat example.com-queue.service

[Unit]
Description=Laravel queue worker

[Service]
User=www-data
Group=www-data
Restart=on-failure
ExecStart=/usr/bin/php /var/www/html/example.com/web/artisan queue:work --daemon

[Install]
WantedBy=multi-user.target

The website https://www.freedesktop.org/software/systemd/man/systemd.service.html#Restart= states, that on-failure does not restart "on success".

If we look closer at the service state, we see that the Laravel Queue is actually exiting with a "0" exit code:

# systemctl status example.com-queue.service
● example.com-queue.service - Laravel queue worker
     Loaded: loaded (/etc/systemd/system/example.com-queue.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Sat 2021-08-14 15:11:18 CEST; 2 days ago
    Process: 2417820 ExecStart=/usr/bin/php /var/www/html/example.com/web/artisan queue:work --daemon (code=exited, status=0/SUCCESS)
Aug 14 15:11:08 example php[2417820]: [2021-08-14 15:11:08][16768] Processing: App\Notifications\UploadSuccess
Aug 14 15:11:18 example php[2417820]: [2021-08-14 15:11:18][16768] Failed:     App\Notifications\UploadSuccess
Aug 14 15:11:18 example systemd[1]: example.com-queue.service: Succeeded.

There is a easy fix: We could switch to "Restart=always", but I want to shed light on the fact and find out why Laravel - in case of an error - returns a zero exit code. Also we are afraid that "Restart=always" might cause unwanted side effects.

Answer 1

Usually, I use supervisor to handle queues. However, it's very advanced, and there are many article/support published on its configuration, and you can find them easily.

I'm following this supervisor configuration in the Laravel article

That's perfectly worked for me, and I've configured a lot of Laravel-projects queues by it. It also has many built-in options to handle queue-failure, restarting etc. Moreover, We can run multiple workers using it easily in minimum time. Its process handling approach is very optimized and well documented.

Basic Installation process:

Manage Laravel queues or any managed process which needs to be run on the server

sudo apt-get install supervisor

load directory cd /etc/supervisor/conf.d and create a configuration file my-project-worker.conf and add the following configurations

[program:<custom-worker-name>]  
command=<project-directory-path>/artisan queue:work --tries=3  --sleep=3
directory=<project-directory-path i.e /var/www/html/example-project>
stdout_logfile=<custom-path-to-store-log i.e /var/www/html/example-project/storage/logs/supervisord.log>
redirect_stderr=true
autostart=true
autorestart=true
numprocs=1
stopwaitsecs=3600

## For Multiple Workers ##

[program:%(<custom-worker-name>)s_%(process_num)02d]  
command=<project-directory-path>/artisan queue:work --tries=3  --sleep=3
directory=<project-directory-path i.e /var/www/html/example-project>
stdout_logfile=<custom-path-to-store-log i.e /var/www/html/example-project/storage/logs/supervisord.log>
redirect_stderr=true
autostart=true
autorestart=true
numprocs=3
stopwaitsecs=3600

Note: Make sure set <custom-worker-name>, <project-directory-path> and <custom-path-to-store-log> according to your project.

Remember that when you deploy new code which contains modifications to existing job code, you need to restart the supervisor process or that particular job.

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start <custom-worker-name>:*

To check running supervisor processes:

sudo supervisorctl

To check the status of the process:

sudo systemctl status supervisor

To stop the process:

sudo systemctl stop supervisor