5

I'm managing audio conversions with Laravel Queues and beanstalkd, monitored by supervisord.

When a user upload an audio file, it goes to AudioController.php that triggers a Queue::push('AudioProcess'), that itself triggers an exec('sh some_script.sh some_audio.mp3') to process the audio and set the application Audio model status to 1 when it's done.

I did a bunch of uploads to test, here are the records Audio upload records

1 means the AudioProcess.php worker has been executed and 0 means the AudioProcess.php worker hasn't been executed.

I guess it might come from either Laravel Queues management or beanstalkd, but I can't find anything relevant in the logs (laravel.log, my supervisord queue.log, php_errors.log).

I'm running a staging and a production environment on the same server, so there are two Laravel applications and therefore two php artisan queue:listen commands running at the same time (each with --env specified), if it has something to do with my issue. It worked well few weeks ago, then I dropped the project for a while and recaught it lately. I did some apt-get update && apt-get upgrade too.

Why is Laravel or beanstalkd not processing all jobs?

ryancey
  • 1,037
  • 2
  • 11
  • 27
  • How did you setup your supervisor, share some code, is that running at all? – The Alpha Nov 10 '14 at 03:20
  • supervisord is running with the right configuration loaded, and the daemons it should be running (`php artisan queue:listen --env=[...]` for `staging` and `production`) are running. The problem comes from either Laravel Queues or beanstalkd that don't work all the jobs. I any case it's useful to you, here are the [supervisord main configuration](http://pastebin.com/WR3qgQFb) and the [staging configuration](http://pastebin.com/EY79WLQP). – ryancey Nov 10 '14 at 07:36
  • Also, [here is a dump](http://pastebin.com/5ZpAr094) of a `ps afx` concerning `supervisord` and `php`. – ryancey Nov 10 '14 at 08:02
  • I didn't find any `[program]` section in your `supervisor config file`. Also, did you add the process using `add ` command from terminal/command prompt? – The Alpha Nov 10 '14 at 18:56
  • See the staging configuration (second link). Yes it has been added correctly and is running as you can see in the `ps afx` dump. – ryancey Nov 10 '14 at 21:01
  • Try to run the job manually from command prompt/terminal using `queue:work` after you add it to the queue and observe the console. – The Alpha Nov 10 '14 at 21:03

1 Answers1

4

The issue was likely happening because the two php artisan queue:listen (for each environment) were sending jobs to the same beanstalkd queue (default). It resulted in some kind of conflict, but I can't tell more...

I fixed it by making each environment's php artisan queue:listen send jobs to a [environment] queue. Therefore I have now two queues, staging and production.

How to do it

  • Create a app/config/[environment]/queue.php that overwrites the [connections => [beanstalkd => [queue]]] parameter from default to the name of that environment.

  • Specify the name of the queue to Artisan: php artisan queue:listen --queue=[environment]

ryancey
  • 1,037
  • 2
  • 11
  • 27
  • 1
    Thanks, was loosing my mind with this! I had three sites on the same server and jobs were getting picked up by the wrong queues – th3hamburgler May 13 '15 at 09:02