FFMpeg + Beanstalk: How to pass the processes to it or achieve the same result without using Beanstalk

Question

My problem is that FFMpeg and Mencoder are extremely resourceful and running even one process of it makes HTTPd slow down but multiple processes of (FFMPEG /Mencoder) just hang it (HTTPd) completely. I would like my conversions to be processed with Beanstalk, for example.

My concrete question is: How to transfer my current jobs to Beanstalk?

I have a simple PHP code that triggers conversion:

RunInBackground('convert.php', array($upload, $video_id), $log_path);

Now what would Beanstalk correct code would look like so these processes would NOT start all at the same time if multiple videos are uploaded?

If you believe that for my needs is better to use something else but Beanstalk and you know how to implement it, I would be still happy to see it!

Thanks in advance,
Ilia

score 3 · Accepted Answer · answered Jul 13 '12 at 16:48

3

Two possibilities:
- try xuggler for your conversions - runs inline instead of having to spawn jobs
- make a 'file process queue' using a db or file.. having your conversion process just query this for 'outstanding files to process' but only have one of these running at a time. It would run independantly of your main task but could post its status somewhere that your main job could read. EG "Busy" or "3 files in queue" or "available"

answered Jul 13 '12 at 16:48

ethrbunny

10,379
9
69
131

you think it's better to use db queue rather than Beanstalk or Resque? – Ilia Ross Jul 13 '12 at 17:26
Since it sounds like part of your problem is controlling how many instances of Beanstalk/etc are running.. that's why I suggested a separate mechanism for the conversion that would maintain one instance. I don't know BeanStalk or Resque but it sounds like you are resource limited. – ethrbunny Jul 13 '12 at 20:12

score 1 · Answer 2 · answered Jul 14 '12 at 07:52

Based on the great idea provided by ethrbunny, I decided to achieve my goal with out using Beanstalk or Resque! After all I was more than happy! I bet it will be useful for all of those who is trying to limit FFMpeg / MPlayer / MEncoder processes to the only one active process on the server to be able to keep other processes running well, such as httpd for example.

Step by step!

Create a database, I called it "encoding_queue":

 CREATE TABLE IF NOT EXISTS `encoding_queue` (
`queue_timestamp` int(11) NOT NULL DEFAULT '0',
`uploaded_video_id` int(11) NOT NULL DEFAULT '0',
`uploaded_file_name` varchar(255) NOT NULL DEFAULT '',
`log_file_path` varchar(255) NOT NULL DEFAULT '',
PRIMARY KEY (`queue_timestamp`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;

In your main upload file, right where you trigger conversion process:

// Let's check if ffmpeg, mplayer, mencoder or mediainfo processes are running //   
exec("ps ax | grep -v grep | grep ffmpeg", $ffmpeg);
exec("ps ax | grep -v grep | grep mplayer", $mplayer);
exec("ps ax | grep -v grep | grep mencoder", $mencoder);
exec("ps ax | grep -v grep | grep mediainfo", $mediainfo);

if(empty($ffmpeg) && empty($mplayer) && empty($mencoder) && empty($mediainfo)) {
// As non of them are running we start conversion process //

   RunInBackground('convert.php', array($uploaded_file_name , $uploaded_video_id ), $log_file_path);

} else {

// As the ffmpeg or mplayer or mencoder or mediainfo is running we add the process to the queue into the database//

   //Connecting to database using ADOdb //             

   $sql = "INSERT INTO encoding_queue SET queue_timestamp = '" .time(). "', 
   uploaded_video_id = '" .mysql_real_escape_string($uploaded_video_id ). "',
   uploaded_file_name = '" .mysql_real_escape_string($uploaded_file_name ). "',
   log_file_path = '" .mysql_real_escape_string($log_file_path). "'";
   $conn->execute($sql);
}

After we added values to the database we need to check if the previous conversion is finished and we ready to start the new one! Create convert_queue.php file and add it to your CRON jobs, let's say every 5 minutes!

<?php
require('path/to/your/config.php');
require('path/to/your/function.php');

exec("ps ax | grep -v grep | grep ffmpeg", $ffmpeg);
exec("ps ax | grep -v grep | grep mplayer", $mplayer);
exec("ps ax | grep -v grep | grep mencoder", $mencoder);
exec("ps ax | grep -v grep | grep mediainfo", $mediainfo);

$sql = "SELECT * FROM encoding_queue ORDER BY queue_timestamp ASC LIMIT 1";
$rs = $conn->execute($sql);
$queue = $rs->getrows();
if ( $conn->Affected_Rows() == 0 ) {
die ("Error! There is no records in the database. Nothing to convert. Stopping...");
}
if (file_exists($config['VDO_DIR'] . '/' .$queue[0]['uploaded_file_name'])) {
  if(empty($ffmpeg) && empty($mplayer) && empty($mencoder) && empty($mediainfo)) {
     RunInBackground('convert.php', array($queue[0]['uploaded_file_name'], $queue[0]['uploaded_video_id']), $queue[0]['log_file_path']);
} else {
     die ("Another conversion process is still in progress. Stopping...");
 }
  } else { 
     die ("Error! There is no video file ".$queue[0]['uploaded_file_name']." found. Nothing to convert. Stopping..."); }
 ?>

Finally as we processed the first entry in the queue, we need to remove it from database so the convert_queue.php could start working on the other one. Run it somewhere on your cleanup or add it to convert_queue.php, but in case the process fails you will not be able to restart it again:
```
$sql = "DELETE FROM encoding_queue WHERE uploaded_video_id = '" .mysql_real_escape_string($uploaded_video_id ). "' LIMIT 1";
$conn->execute($sql);
```

So what do we get? If the file that was uploaded is the only one at the moment and there is no more conversions happening at the moment on the server than we start conversion process! If there is a video file that's already being converted than we put the information about the uploaded file into the database and then we wait for the moment to start converting it right when the conversion of the other file is over.

Queue is the real queue - files are being processes based on the upload time!

This is a FFMpeg / MPlayer / MEncoder queue solution!

All the best, Ilia

FFMpeg + Beanstalk: How to pass the processes to it or achieve the same result without using Beanstalk

2 Answers2