8

I am working through the GNU Parallel totorial. In the "More than one argument" section, there is the following example (note: num30000 is a text file with numbers 1 to 30,000 on sequential lines):

For better parallelism GNU Parallel can distribute the arguments between all the parallel jobs when end of file is met.

Running 4 jobs in parallel will split the last line of arguments into 4 jobs resulting in a total of 5 jobs:

  cat num30000 | parallel --jobs 4 -m echo | wc -l

Output:

  5

My question is: why do we expect 5 total jobs? I am clearly missing a point, although I don't know if it's important. I expected 4 jobs since 30,000 is divisible by 4. I decided to post this question after running the following:

cat num30000 | parallel --jobs 4 -m echo | colrm 12

which results in:

1 2 3 4 5 6
23696 23697
25273 25274
26850 26851
28427 28428

This looks to me like the first echo command is passed the first 23,695 arguments. Then, the remaining are split into 4 more jobs with argument counts of 1577, 1577, 1577, and 1574. Am I misunderstanding what the call to parallel is supposed to do? Thank you!

Community
  • 1
  • 1
Steve Koch
  • 912
  • 8
  • 21

2 Answers2

7

Your price for this answer is to help me improve the example in a way so that you would have understood it in the first reading.

So what happens is the following:

GNU Parallel figures out that the limit is 131071. Then it sees how many jobs are currently running (0). Is that less than the the number of jobs to run in parallel (4): Then it reads arguments up to the 131071 limit and start that job. This is the first job.

Now GNU Parallel reads arguments again. This time it reads all the rest and hits end of file. "Oh," GNU Parallel thinks. "If this is the end of file, then I will spread all the arguments over all job slots (4)." So it takes all the rest of the arguments and spreads them over 4 jobs. It then starts 3 of the jobs. 4 jobs are now running.

One of the running jobs finishes, giving a free job slot; so GNU Parallel starts the final job.

The reason for this design is more clear if you have 4 cores and 100 arguments: The 100 arguments would easily fit on a single line, but normally running 4 jobs with 25 args on a 4 core machine will be faster than running 1 job with 100 args.

Ole Tange
  • 31,768
  • 5
  • 86
  • 104
  • 1
    I am happy to pay the penalty, I mean price :) However, I may still not have enough understanding to do so. Here's an attempt: "`Requesting` 4 jobs in parallel will split the last line of arguments into 4 jobs. Since `cat num30000 | parallel -m echo` creates two lines of arguments, the first job is started on the first, maximal length line (up to line 23695). The second line has the EOF marker, so GNU Parallel spreads those arguments as evenly as possible over the next 4 jobs, for a total of 5 jobs. ... compare to the command `head -n 20000 num30000 | parallel --jobs 4 -m echo | wc -l`" – Steve Koch Jan 13 '14 at 17:39
3

I am understanding the language now. The -m parameter asks parallel to put as many arguments as possible onto the command line. My limit of 131071 characters means two total echo commands are generated. The first one has up to number 23695. The second has the rest. The --jobs 4 parameter only affects the second command. This is what the tutorial means by "last line of arguments." So, I understand why there are 5 total jobs. However, I don't understand why --jobs only affects the last line of arguments, but that is not the question I asked.

Steve Koch
  • 912
  • 8
  • 21