3

I have "randomly" some jobs failing in my beanstalked queue. Two questions:

1/ By looking at the failed job table in the DB how can I interprate what is logged inside? The Data closure means "nothing" when reading it raw. Is there something to do to have more info?

2/ I went through the console logs of Laravel and this is what fails: 

[2015-04-29 15:40:51] production.ERROR: exception 'ErrorException' with message 'Trying to get property of non-object' in /home/forge/api.hello.me/vendor/jeremeamia/SuperClosure/src/Jeremeamia/SuperClosure/SerializableClosure.php(99) : eval()'d code:6
Stack trace:...

However I have no clue what exactly is failing... the stacktrace doesn't help and just starts at 

    [internal function]: Jeremeamia\SuperClosure\SerializableClosure::{closure}(Object(Illuminate\Queue\Jobs\BeanstalkdJob))
#2 /home/forge/api.hello.me/vendor/jeremeamia/SuperClosure/src/Jeremeamia/SuperClosure/SerializableClosure.php(64): ReflectionFunction->invokeArgs(Array)

and finished a few lines later at 

#15 /home/forge/api.hello.me/vendor/symfony/console/Symfony/Component/Console/Application.php(124): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
    #16 /home/forge/api.hello.me/artisan(59): Symfony\Component\Console\Application->run()
    #17 {main} [] []

Any idea on how I could understand / find out what is happening? The queue is not even failing all the time...

ps: full stack trace:

#0 /home/forge/api.hello.me/vendor/jeremeamia/SuperClosure/src/Jeremeamia/SuperClosure/SerializableClosure.php(99) : eval()'d code(6): Illuminate\Exception\Handler->handleError(8, 'Trying to get p...', '/home/forge/api...', 6, Array)
#1 [internal function]: Jeremeamia\SuperClosure\SerializableClosure::{closure}(Object(Illuminate\Queue\Jobs\BeanstalkdJob))
#2 /home/forge/api.hello.me/vendor/jeremeamia/SuperClosure/src/Jeremeamia/SuperClosure/SerializableClosure.php(64): ReflectionFunction->invokeArgs(Array)
#3 /home/forge/api.hello.me/vendor/laravel/framework/src/Illuminate/Queue/IlluminateQueueClosure.php(36): Jeremeamia\SuperClosure\SerializableClosure->__invoke(Object(Illuminate\Queue\Jobs\BeanstalkdJob))
#4 /home/forge/api.hello.me/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(96): IlluminateQueueClosure->fire(Object(Illuminate\Queue\Jobs\BeanstalkdJob), Array)
#5 /home/forge/api.hello.me/vendor/laravel/framework/src/Illuminate/Queue/Jobs/BeanstalkdJob.php(50): Illuminate\Queue\Jobs\Job->resolveAndFire(Array)
#6 /home/forge/api.hello.me/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(205): Illuminate\Queue\Jobs\BeanstalkdJob->fire()
#7 /home/forge/api.hello.me/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(157): Illuminate\Queue\Worker->process('beanstalkd', Object(Illuminate\Queue\Jobs\BeanstalkdJob), '3', '0')
#8 /home/forge/api.hello.me/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(105): Illuminate\Queue\Worker->pop('beanstalkd', 'default', '0', '10', '3')
#9 /home/forge/api.hello.me/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(67): Illuminate\Queue\Console\WorkCommand->runWorker('beanstalkd', 'default', '0', '128', false)
#10 /home/forge/api.hello.me/vendor/laravel/framework/src/Illuminate/Console/Command.php(112): Illuminate\Queue\Console\WorkCommand->fire()
#11 /home/forge/api.hello.me/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(253): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /home/forge/api.hello.me/vendor/laravel/framework/src/Illuminate/Console/Command.php(100): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /home/forge/api.hello.me/vendor/symfony/console/Symfony/Component/Console/Application.php(889): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /home/forge/api.hello.me/vendor/symfony/console/Symfony/Component/Console/Application.php(193): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /home/forge/api.hello.me/vendor/symfony/console/Symfony/Component/Console/Application.php(124): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /home/forge/api.hello.me/artisan(59): Symfony\Component\Console\Application->run()
#17 {main} [] []
commandantp
  • 947
  • 1
  • 12
  • 25
  • The stacktrace and error log don't seem to be very helpfull in this case. Could you perhaps post the code being pushed onto the queue? – Bogdan Apr 30 '15 at 10:03
  • by looking at the failed job, you can see the class/function that is being called. print out logs, and find where it crashes... you can repush jobs from the failed queue so you should be able to catch these jobs and reproduce them – NiRR Apr 30 '15 at 10:36
  • We need to see what you are pushing. – Ravan Scafi Apr 30 '15 at 11:06
  • Hey @Ravan, NiRR & Bogdan thank you for getting back! Sorry couldn't reply earlier. So even if I repush it in the queue I have no clue which job is being fired (which controller is it going through etc) the app is in production. Any idea how I could find that out? – commandantp Apr 30 '15 at 16:48

1 Answers1

0

The way Laravel queues work is to serialize all the data you provide as the payload, and it sticks that in your database. When you process the queue, it will magically unserialize all this data so that you have basically the same object in memory as you did when you popped it into the queue. The problem comes in when you change the class definition of a serialized object, or your object does not quite serialize correct (which seems to be the case here)...

I would suggest that you try not serializing anonymous functions, because Laravel uses SuperClosure to do this, and it's pretty hacky.

[internal function]: Jeremeamia\SuperClosure\SerializableClosure::{closure}(Object(Illuminate\Queue\Jobs\BeanstalkdJob))

Try pushing simpler objects into your Beanstalk queue (i.e. don't send objects to the queue) and you won't have this issue.

Danny Kopping
  • 4,862
  • 2
  • 29
  • 38
  • Thanks, I'll try to change that again, strangely the queues only get IDs or a string pushed and then loads the models again once in the queue. – commandantp May 03 '15 at 14:44
  • What does the data look like in the table? – Danny Kopping May 03 '15 at 14:46
  • Just an ID number, the autoincremented one. Ah... A username too, maybe special characters could cause problems? – commandantp May 03 '15 at 15:02
  • Doubt it. Could you paste the SQL dump of the data directly? – Danny Kopping May 03 '15 at 15:19
  • I will try. My other problem is that I have no clue which queue fails so I can't even send the sql dump of that specific problem. How could j track back where it failed? I have many queues in many controllers... – commandantp May 03 '15 at 15:24
  • Are you able to reliably reproduce this error? If so, keep reproducing it and each time try disable or remove other queue jobs until you find the culprit – Danny Kopping May 03 '15 at 20:35
  • Not yet been able to reproduce. Restarted the queue and so far has worked...will keep an eye on that one if it happens again – commandantp May 03 '15 at 21:47