0

I have a push queue set up to dial quite a few phone numbers and play back a recording - a blast announcement system powered by Twilio. It takes time to iterate through each number and place the call, so I am hoping to use a push queue to speed up the navigation of the app.

I attempted to use Iron.io push queues in the past with Laravel 4 and it seems that any task that takes a while to run, or if the HTTP request was slow at first, the code within the fire() method runs multiple times, even with $job->delete()

Here is an example of my queue handler - 

    class callLotsOfPeople {
    public function fire($job, $data) {
        // Do stuff with data, like calling lots of people.. takes time
        $job->delete();
        // For some reason this method can be called multiple times after a single queue push, 
        // resulting in multiple phone calls and angry clients
    }
}
NightMICU
  • 9,000
  • 30
  • 89
  • 121

2 Answers2

2

This might be too late, but I had the same issue. I found a fix and have submitted a pull request to have it included in Laravel 4.1.

Basically, the code changes allow you to pass an $options array like this

Queue::push('MyJob', $message, $queue, array('timeout' => 300));

The functionality is already there in the IronMQ.class.php, but I couldn't find an easy way to pass them in from Laravel. Hopefully they include this, makes the multiple job submission issue go away. :-)

https://github.com/laravel/framework/pull/3555

EDITED: Changed Queue::pull to Queue::push, small typo.

titleistfour
  • 305
  • 3
  • 12
  • Hey, thanks for this! Would I use this in place of `Queue::marshal()`? – NightMICU Feb 14 '14 at 01:00
  • You would still need to marshal the job. Iron.io has a default timeout of 60 seconds. If accepted, this would allow you to set a much higher timeout on long running jobs. When an Iron.io job times out, it is submitted again, thus the problem with multiple jobs. – titleistfour Feb 14 '14 at 02:31
  • I'm a bit confused.. where you have Queue::pull, do you really mean Queue::push? Could you show an example? – NightMICU Feb 14 '14 at 02:36
  • LOL just saw your edit - big typo, my friend! This is awesome, I can use it now? I believe the production app in question is running 4.0 – NightMICU Feb 14 '14 at 02:37
  • The code has not been accepted into Laravel yet. Use at your own risk :-) – titleistfour Feb 14 '14 at 03:17
1

I've yet to leverage Push Queues fully, but a quick glance at IronMQ docs revealed the following:

Long Running Processes - aka 202’s

If you’d like to take some time to process a message, more than the 60 second timeout, you must respond with HTTP status code 202. Be sure to set the “timeout” value when posting your message to the maximum amount of time you’d like your processing to take. If you do not explicitly delete the message before the “timeout” has passed, the message will be retried. To delete the message, check the “Iron-Subscriber-Message-Url” header and send a DELETE request to that URL.

via: http://dev.iron.io/mq/reference/push_queues/#long_running_processes__aka_202s

Now, the timeout isn't something that Laravel seems to support at the moment since the payload is created behind the scenes with no easy access. You can create a Pull Request on the 4.1 branch to implement this functionality specifically for Iron push queues (tip: you'd need to edit both QueueInterface and all Queue drivers' push() function).

As a work-around, maybe you can just $job->delete() from the get-go (rather than after the time-consuming task) and just Queue::push() it again (or some chunk of it) if there are errors? Something like:

    class callLotsOfPeople {
        public function fire($job, $data) {
            $job->delete();
            
            // Do stuff with data, like calling lots of people.. takes time
    
            if ($error) {
                    Queue::push(...);
            }
        }
    }

Let me know how it goes, I may have a similar situation in the future and would like to know how you solve it!

Community
  • 1
  • 1
whycodewhyyyy
  • 160
  • 1
  • 8
  • The problem struck again today, this time with the default 'sync' queue in Twilio. A script was trying to pull phone numbers from a database and hit a "Trying to get property of non-object" error on one number, the $job->delete() was after the `foreach` so it repeated hundreds of times, every time calling phone numbers. Laravel should really advise that $job->delete() should be the first line. – NightMICU Jan 09 '14 at 20:00
  • 1
    Well, it's not always the case that `$job->delete()` would be best placed as the first line. In your situation it makes sense, as it does in mine, but there may be cases where placing it at the end makes more sense. Regardless, glad this helped! :) – whycodewhyyyy Jan 10 '14 at 18:01