0

I am working on creating a video timelapse. All the photos I took are .jpg images shot at 4:3 aspect ratio. 2592x1944 resolution. I want them to be 16:9 at 1920x1080.

I have written a little script to do this, as well as piece together a video, but I am running into problems!

This script works great when I have less than 300 files, but when I have more than that, the "convert" command I use to crop images just eats up all the memory and freezes up my computer (with 8GB RAM), all without cropping a single photo!

There are grand total about 300,000 photos... which I plan to do in batches of around 50,000 photos each. So this is obviously a problem that must be overcome. But for now I am testing with about ~700 photos.

Here is the script I have written. I am running Ubuntu 14.04

Note that I have identified the problem as occurring during the "convert" function. It still happens if I run it outside of the rest of the script.

mkdir resized

echo Begin Resizing!
mogrify -path resized -resize 1920x1440! *.JPG #Resizes all files, maintaining 4:3 aspect ratio

echo Resizing Complete! Begin Cropping!
cd resized
convert *.JPG -crop 1920x1080+0+$1 D$2P$3 #Crops all 4:3 files to 16:9 aspect ratio. Takes command line arguments for cropping dimensions and filename (Memory leaks, can't crop ? 300 images!)

echo Cropping Complete!
mkdir cropped
mv D* cropped/ #Moves all cropped photos into new directory
cd cropped
find . -type f -exec mv '{}' '{}'.JPG \; #Adds '.JPG' to each filename
ls *.JPG -1tr > files.txt #Creates List of files

echo MAKING VIDEO!
mencoder -nosound -noskip -oac copy -ovc copy -o output.avi -mf fps=30 'mf://@files.txt' #Creates .avi video from jpg images (very fast)
#avconv -i output.avi -c:v libx264 -preset slow -crf 15 output-final.mkv #Converts .avi to .mkv video (Very Slow - looking for better method than making 2 vids)

echo ---------
echo ALL DONE! 
echo ---------

I played around with it and found that 300-350 images seems to be where it freezes the computer up entirely.

This issue is my biggest problem - but there are two more problems which are somewhat related. Solving the first one may involve bypassing this issue.

  1. I currently resize each photo, save it as a new file, and then crop it, saving it as yet another file. Since I am going to be working with ~50,000 photos at once... this will get cumbersome. Anyway I can crop and resize a photo all at once? It is -crucial- however, that I be able to specify the exact place where the photos is cropped. I can not just crop equal amounts off the top and bottom.

  2. I am currently making it into a video in the only process I could get to work - making a .avi file from the jpgs, and then making a second video converted from this one. This is also an obvious unnecessary step, but I'm not sure how to get directly to a .mkv (or better, a .mp4). This problem may end up being moot, as I am looking into non-terminal solutions to make the actual video from the photos (namely Sony Vegas and GoPro Studio).

Brian C
  • 1,333
  • 3
  • 19
  • 36
  • You aren't going to be able to process tens of thousands of file in one go anyway, because there are limits on how long the command line can be. I would rewrite the inner loop to run on a single file at a time, then figure out if I can run multiple instances in batch. Then the temporary files can be cleaned up within the loop, too; so you will end up spending a lot less hard drive space on temporary files. – tripleee Mar 15 '15 at 04:52
  • I now have this as a loop: "for f in *.JPG do echo "mog $f" mogrify -path cropped -resize 1920x1440! $f echo "convert $f" convert cropped/$f -crop 1920x1080+0+$1 -set filename:name '%t' cropped/'%[filename:name].JPG' done " Would this be able to do 50k files? I'm running it now on 700 and the memory usage is holding steady at 55%, CPU at 85%. It would take a while... but I assume possible. – Brian C Mar 15 '15 at 07:41

2 Answers2

0

Replace *.JPG with a loop for your files so that convert exits every time. That way you avoid the issue.

Example (untested):

#!/bin/bash
FILES=/path/to/*.JPG
for f in $FILES
do
    convert $f -crop 1920x1080+0+$1 D$2P$3
done
Norbert
  • 6,026
  • 3
  • 17
  • 40
  • OK. What you typed did not work, but I managed to tweak it to work. I have: "for f in ls *.JPG". For some reason using variable FILES led to getting a "Command Not Found: File1.JPG" error. Also - Once I fixed that, it began automatically writing over each file with each iteration of the loop. So I fixed that by using "convert $f -crop 1920x1080+0+87 -set filename:name '%t' cropped/'%[filename:name].JPG'" so it writes to a new directory. This also eliminates the need to move the files and rename them all, as I have been doing. – Brian C Mar 15 '15 at 07:09
  • `for f in ls *.JPG` does exactly the same thing, except it adds a (presumably nonexistent) file `ls` to the start of the list of tokens to loop over. Don't use the `ls` command for this; it is wasteful and [sometimes even wrong](http://mywiki.wooledge.org/ParsingLs). – tripleee Mar 15 '15 at 07:59
  • However, I also agree that the `FILES` variable is useless and potentially a complication. – tripleee Mar 15 '15 at 08:01
0

Maybe it helps to set some limits on memory and define a temporary path.

I did about 1200 images this way without any problems.

convert -monitor -limit memory 2GiB -limit map 4GiB -define registry:temporary-path=/data/tmp *.JPG -crop 1920x1080+0+$1 D$2P$3

http://www.imagemagick.org/script/command-line-options.php#limit

http://www.imagemagick.org/script/command-line-options.php#define

FireMagick
  • 1
  • 2