Running programs in parallel using xargs

Question

I currently have the current script.

#!/bin/bash
# script.sh

for i in {0..99}; do
   script-to-run.sh input/ output/ $i
done

I wish to run it in parallel using xargs. I have tried

script.sh | xargs -P8

But doing the above only executed once at the time. No luck with -n8 as well. Adding & at the end of the line to be executed in the script for loop would try to run the script 99 times at once. How do I execute the loop only 8 at the time, up to 100 total.

That is what I initially wanted to do, but had to resort to xargs because I am on Windows. I was not able to get GNU Parallel running on Windows — Olivier, Feb 06 '15 at 03:21
Is that script calling itself or did you just confuse the names when you asked here? — Etan Reisner, Feb 06 '15 at 03:24
The answer to https://stackoverflow.com/questions/3321738/shell-scripting-using-xargs-to-execute-parallel-instances-of-a-shell-function is relevant here. — Etan Reisner, Feb 06 '15 at 03:28

score 177 · Accepted Answer · answered Feb 06 '15 at 03:26

177

From the xargs man page:

This manual page documents the GNU version of xargs. xargs reads items from the standard input, delimited by blanks (which can be protected with double or single quotes or a backslash) or newlines, and executes the command (default is /bin/echo) one or more times with any initial- arguments followed by items read from standard input. Blank lines on the standard input are ignored.

Which means that for your example xargs is waiting and collecting all of the output from your script and then running echo <that output>. Not exactly all that useful nor what you wanted.

The -n argument is how many items from the input to use with each command that gets run (nothing, by itself, about parallelism here).

To do what you want with xargs you would need to do something more like this (untested):

printf %s\\n {0..99} | xargs -n 1 -P 8 script-to-run.sh input/ output/

Which breaks down like this.

printf %s\\n {0..99} - Print one number per-line from 0 to 99.
Run xargs
- taking at most one argument per run command line
- and run up to eight processes at a time

answered Feb 06 '15 at 03:26

Etan Reisner

77,877
8
106
148

9

Actually you don't need to put the arguments on separate lines; xargs word-splits. So `echo {0..99} |` would work just as well. `<<<{0..99}` doesn't seem to work; although `<< – rici Feb 06 '15 at 03:41
1

@rici Looks like a documentation bug then especially since the documentation for Here Documents *doesn't* mention brace expansion (and it doesn't happen there either in a quick test) though they also don't mention tilde expansion (which doesn't happen for `<<` but does for `<<<` so `*shrug*`). The expansions that do and don't happen in here docs and here strings are a bit odd to my mind. – Etan Reisner Feb 06 '15 at 03:49
1

How can you separate results from different runs with e.g. newlines? – nirvana-msu Oct 08 '17 at 00:48
7

Demo: `time head -12 <(yes "1") | xargs -n1 -P4 sleep` will run 12 `sleep 1` commands, 4 parallel. The command will take 3 seconds. – Walter A Oct 16 '19 at 10:13
3

It's probably worth noting that `-P 0` will use the number of cpus on the system – slf Dec 03 '21 at 17:35
You could also use `seq 0 99` to get the numeric sequence (with newline separators; works at least on linux). – jena Jan 06 '23 at 13:09

score 12 · Answer 2 · answered Jul 23 '22 at 21:36

12

Here's an example running commands in parallel in conjuction with find:

find -name "*.wav" -print0 | xargs -0 -t -I % -P $(nproc) flac %

-print0 terminates filenames with a null byte rather than a newline so we can use -0 in xargs to prevent filenames with spaces being treated as two seperate arguments.

-t means verbose, makes xargs print every command it's executing, can be useful, remove if not needed.

-I % means replace occurrences of % in the command with arguments read from standard input.

-P $(nproc) means run a maximum of nproc instances of our command in parallel (nproc prints the number of available processing units).

flac % is our command, the -I % from earlier means this will become flac foo.wav

Running programs in parallel using xargs

3 Answers3

Linked