Is there a way to run one job many times using GNU parallel?

Question

I can see how easy it is to run a parallel job on multiple input but is there no other way to run the same job in parallel multiple times with putting the command in a file and repeating it many times?

parallel -j+0 ::: './dosomejob.sh'

but tell parallel to run that job x amount of times in parallel using the number of available cores?

Ole Tange · Answer 1 · 2019-10-24T18:15:43.603

12

If you have less than 10 CPU threads:

parallel -N0 ./dosomejob.sh ::: {1..10}

-N0 = insert 0 arguments.

If you want GNU Parallel to generate combinations, but you want all combinations repeated 5 times try:

parallel mycommand {1} {2} {3} {4} fixed_arg1 fixed_arg2 \
  :::: arg_file1.txt  arg_file2.txt \
  ::::+ arg_file3.txt  arg_file4.txt \
  ::: {1..5}

edited Oct 24 '19 at 18:15

answered Jul 15 '18 at 03:43

Ole Tange

31,768
5
86
104

I don't quite get the `-N0`. Can it be omitted? – stefanbschneider Oct 24 '19 at 07:54
1

If you do that, you will pass argument 1 though 10 to `dosomejob.sh`. If you do not use that argument for anything, then yes you can omit `-N0`. – Ole Tange Oct 24 '19 at 07:56
In my case, I have `parallel command :::: arg_file1.txt :::: arg_file2.txt ::::+ arg_file3.txt ::::+ arg_file4.txt ::: "fixed arg" ::: "fixed arg" ::: {1..5}`. The `::: {1..5}` is supposed to repeat the command 5x. However, it breaks the command. I tried setting `-N6` to only consider the first 6 args and ignore the 1-5, but it still breaks. Am I missing something? – stefanbschneider Oct 24 '19 at 09:32
@CGFoX Yes. The {1} {2} {3} {4} in the example. Try using `--dry-run` to see what will actaully be run. – Ole Tange Dec 13 '20 at 10:21
fish shell: `parallel -N0 ./dosomejob.sh ::: (seq 10)` – Matěj Šmíd Nov 24 '22 at 07:42

score 4 · Accepted Answer · edited Jul 10 '18 at 10:35

4

Typical. I spend ages trying to work this out and when I post a question, I think I actually worked it out.

parallel -j+0 './dosomejob.sh' ::: {1..10}

Is this the correct way to do this? Is there a better parallel way?

edited Jul 10 '18 at 10:35

user7761803

209
2
9

answered Jul 09 '18 at 13:36

Jon Scobie

490
4
10

1

Rubber duck effect in action. :) – Sergio Tulentsev Jul 09 '18 at 13:41
`-j+0` is default and can be left out. – Ole Tange Jan 04 '19 at 22:41

Is there a way to run one job many times using GNU parallel?

2 Answers2