GNU Parallel not completing all "find" commands

Question

I'm trying to parallelize a command without success, there are about 100 video files to convert.

This format resulted in 8 files being converted in a random order:

parallel -j 8 videoConvert ::: `find "/mnt/bulk-data-02/videos/convert" -iname "*.mov"`

This format resulted in 12 files being converted in a random order:

find "/mnt/bulk-data-02/videos/convert" -iname "*.mov" | parallel -j 8 videoConvert

Please Help, Thanks.

The results of: find "/mnt/bulk-data-02/videos/convert" -iname "*.mkv" | parallel --dryrun -j 8 videoConvert

Are as follows, but I had to delete some because the website doesn't like large code blocks:

videoConvert /mnt/bulk-data-02/videos/convert/053.mkv
videoConvert /mnt/bulk-data-02/videos/convert/005.mkv
videoConvert /mnt/bulk-data-02/videos/convert/100.mkv
videoConvert /mnt/bulk-data-02/videos/convert/112.mkv
videoConvert /mnt/bulk-data-02/videos/convert/032.mkv
videoConvert /mnt/bulk-data-02/videos/convert/025.mkv
videoConvert /mnt/bulk-data-02/videos/convert/101.mkv
videoConvert /mnt/bulk-data-02/videos/convert/033.mkv
videoConvert /mnt/bulk-data-02/videos/convert/054.mkv
videoConvert /mnt/bulk-data-02/videos/convert/039.mkv
videoConvert /mnt/bulk-data-02/videos/convert/004.mkv
videoConvert /mnt/bulk-data-02/videos/convert/060.mkv
videoConvert /mnt/bulk-data-02/videos/convert/041.mkv
videoConvert /mnt/bulk-data-02/videos/convert/038.mkv
videoConvert /mnt/bulk-data-02/videos/convert/076.mkv
videoConvert /mnt/bulk-data-02/videos/convert/030.mkv
videoConvert /mnt/bulk-data-02/videos/convert/040.mkv
videoConvert /mnt/bulk-data-02/videos/convert/105.mkv
videoConvert /mnt/bulk-data-02/videos/convert/056.mkv
videoConvert /mnt/bulk-data-02/videos/convert/094.mkv
videoConvert /mnt/bulk-data-02/videos/convert/012.mkv
videoConvert /mnt/bulk-data-02/videos/convert/043.mkv
videoConvert /mnt/bulk-data-02/videos/convert/029.mkv
videoConvert /mnt/bulk-data-02/videos/convert/010.mkv
videoConvert /mnt/bulk-data-02/videos/convert/088.mkv
videoConvert /mnt/bulk-data-02/videos/convert/091.mkv
videoConvert /mnt/bulk-data-02/videos/convert/083.mkv
videoConvert /mnt/bulk-data-02/videos/convert/086.mkv
videoConvert /mnt/bulk-data-02/videos/convert/022.mkv
videoConvert /mnt/bulk-data-02/videos/convert/050.mkv
videoConvert /mnt/bulk-data-02/videos/convert/087.mkv
videoConvert /mnt/bulk-data-02/videos/convert/098.mkv
videoConvert /mnt/bulk-data-02/videos/convert/095.mkv
videoConvert /mnt/bulk-data-02/videos/convert/008.mkv
videoConvert /mnt/bulk-data-02/videos/convert/015.mkv
videoConvert /mnt/bulk-data-02/videos/convert/064.mkv
videoConvert /mnt/bulk-data-02/videos/convert/080.mkv
videoConvert /mnt/bulk-data-02/videos/convert/069.mkv
videoConvert /mnt/bulk-data-02/videos/convert/057.mkv
videoConvert /mnt/bulk-data-02/videos/convert/037.mkv
videoConvert /mnt/bulk-data-02/videos/convert/072.mkv
videoConvert /mnt/bulk-data-02/videos/convert/055.mkv
videoConvert /mnt/bulk-data-02/videos/convert/102.mkv
videoConvert /mnt/bulk-data-02/videos/convert/051.mkv
videoConvert /mnt/bulk-data-02/videos/convert/066.mkv
videoConvert /mnt/bulk-data-02/videos/convert/036.mkv
videoConvert /mnt/bulk-data-02/videos/convert/099.mkv
videoConvert /mnt/bulk-data-02/videos/convert/089.mkv
videoConvert /mnt/bulk-data-02/videos/convert/063.mkv
videoConvert /mnt/bulk-data-02/videos/convert/103.mkv
videoConvert /mnt/bulk-data-02/videos/convert/062.mkv
videoConvert /mnt/bulk-data-02/videos/convert/067.mkv
videoConvert /mnt/bulk-data-02/videos/convert/096.mkv
videoConvert /mnt/bulk-data-02/videos/convert/075.mkv
videoConvert /mnt/bulk-data-02/videos/convert/003.mkv
videoConvert /mnt/bulk-data-02/videos/convert/049.mkv
videoConvert /mnt/bulk-data-02/videos/convert/007.mkv
videoConvert /mnt/bulk-data-02/videos/convert/016.mkv
videoConvert /mnt/bulk-data-02/videos/convert/014.mkv
videoConvert /mnt/bulk-data-02/videos/convert/073.mkv
videoConvert /mnt/bulk-data-02/videos/convert/084.mkv
videoConvert /mnt/bulk-data-02/videos/convert/021.mkv
videoConvert /mnt/bulk-data-02/videos/convert/093.mkv
videoConvert /mnt/bulk-data-02/videos/convert/074.mkv
videoConvert /mnt/bulk-data-02/videos/convert/061.mkv
videoConvert /mnt/bulk-data-02/videos/convert/017.mkv
videoConvert /mnt/bulk-data-02/videos/convert/078.mkv
videoConvert /mnt/bulk-data-02/videos/convert/071.mkv
videoConvert /mnt/bulk-data-02/videos/convert/081.mkv
videoConvert /mnt/bulk-data-02/videos/convert/011.mkv
videoConvert /mnt/bulk-data-02/videos/convert/070.mkv
videoConvert /mnt/bulk-data-02/videos/convert/042.mkv
videoConvert /mnt/bulk-data-02/videos/convert/106.mkv
videoConvert /mnt/bulk-data-02/videos/convert/013.mkv
videoConvert /mnt/bulk-data-02/videos/convert/110.mkv
videoConvert /mnt/bulk-data-02/videos/convert/020.mkv
videoConvert /mnt/bulk-data-02/videos/convert/024.mkv
videoConvert /mnt/bulk-data-02/videos/convert/047.mkv
videoConvert /mnt/bulk-data-02/videos/convert/034.mkv
videoConvert /mnt/bulk-data-02/videos/convert/092.mkv
videoConvert /mnt/bulk-data-02/videos/convert/065.mkv
videoConvert /mnt/bulk-data-02/videos/convert/002.mkv
videoConvert /mnt/bulk-data-02/videos/convert/045.mkv
videoConvert /mnt/bulk-data-02/videos/convert/001.mkv
videoConvert /mnt/bulk-data-02/videos/convert/090.mkv
videoConvert /mnt/bulk-data-02/videos/convert/068.mkv
videoConvert /mnt/bulk-data-02/videos/convert/077.mkv
videoConvert /mnt/bulk-data-02/videos/convert/026.mkv
videoConvert /mnt/bulk-data-02/videos/convert/031.mkv
videoConvert /mnt/bulk-data-02/videos/convert/027.mkv
videoConvert /mnt/bulk-data-02/videos/convert/058.mkv
videoConvert /mnt/bulk-data-02/videos/convert/046.mkv
videoConvert /mnt/bulk-data-02/videos/convert/028.mkv

perhaps [this](https://www.gnu.org/software/parallel/parallel_examples.html#example-calling-bash-functions) — Yarin_007, Jan 29 '23 at 23:12
Using backticks is not recommended: If some of the file names contain space, the shell will split theses, and GNU Parallel will get the wrong input. This probably explains the first issue. For the second command: please update the question with the output of `parallel --dryrun ...`. — Ole Tange, Jan 30 '23 at 08:38
I revised the original post to include. Thanks for the help. — user2434856, Jan 31 '23 at 00:24
When performing these conversions I fire the batch off with nohup and close the terminal. Could this have any interaction with parallel? — user2434856, Jan 31 '23 at 00:31
No matter which way I send the job list to parallel it stops executing additional videoConvert commands once the terminal window is closed. Thanks for helping me confirm the issue actually lies with an incompatibility of parallel with nohup. I will try using tmux and report back. — user2434856, Feb 01 '23 at 16:47

score 1 · Answer 1 · answered Jan 31 '23 at 21:59

If, as your output suggests, all your files are in one directory, you can use:

parallel -j 8 videoConvert ::: /mnt/bulk-data-02/videos/convert/*.mkv

If your files are not in a single directory, you are generally better off piping null-terminated filenames into parallel like this:

find /mnt/... -iname ... -print0 | parallel -0 videoConvert

GNU Parallel not completing all "find" commands

1 Answers1