2

I'm using Beanstalkd message queue for ffmpeg processing video in my project. I use pheanstalk PHP client for beanstalkd queue. My worker looks like:


...
// get latest job
$job = $this->pheanstalk->reserve();
// get the data from the job
$jobData = unserialize($job->getData());
// process job, giving a result
$result = $this->task($jobData);
if ($result) {
  // success = delete
  $this->pheanstalk->delete($job);
} else {
// failed = bury
  $this->pheanstalk->bury($job, 1024);
}
...

In task() method I use smth like:


// video processing
$processVideo = 'ffmpeg -vpre libx264-max -y -i inputfile ...';
shell_exec($processVideo);
// taking screenshots from video
...

As you know, video processing can take long time, and sometimes it exceeds predefined job ttr (time to run). This way the job recieves time out event and comebacks to queue again. I run my worker several times at the same time for asynchronous processing. So the next free worker tries to take unfinished job. Finally, I have two or more processes under one video file. I think, I can define very long ttr, but it is not a good desigion. I've noticed, beanstalkd has DEADLINE_SOON event. But I don't know, how to catch it in my worker. I need it to have an ability to use "touch" command.

Do you have any advices?

Charles
  • 50,943
  • 13
  • 104
  • 142
ASergey
  • 298
  • 3
  • 18

1 Answers1

3

According to the mailing list message (part of this thread), the 'DEADLINE_SOON' message will only be sent back if you have a reserve request being made, when a job is about to have the TTR expire on it.

If you reserve one job, and are processing it (rather than collecting a number of other jobs), you won't see the deadline message though - you aren't looking for it, and if you were, then you would probably not be processing the file in the first place.

Think of the TTR setting as a second chance to get another crack at a job if there is a problem with the worker and it does not touch or delete the job it has picked up. Set the timeout at least as long as you expect the processing to take, and then add a more as a safety margin. Better to wait a while for a complex action than keep failing on the same one over, and over again. If you have set the TTR too low, you'll keep getting the same file, and you'll keep having the TTR expire.

Alister Bulman
  • 34,482
  • 9
  • 71
  • 110
  • 1
    With PHP, I recommend burying the job after you've reserved it, especially in cases where you can't predict the TTR. It's safer than deleting the job and you don't have to worry about touching it (which would be difficult to do in PHP anyway). – kouton Apr 26 '14 at 06:11