Run bash command in parallel for N-processes in batch

Question

I have a command:

python test.py -i input_file -p parameter_1 -o output_file

I want to run this for multiple input files, 4 input files at a time (in parallel) and once they finish run the next 4 and so on using a bash script.

I am not able to find the right answer.

I am trying the following but I am not sure if it is right

for (( n=0; n<12; n++ ))
do
    ((j=j%4)); ((j++==0)) && wait
    python test.py -i input_list[n] -p parameter_1 -o output_$n &
done

Thanks in advance!

If there are 4 running, and _one_ finishes, do you want to start another one, or wait _for all 4_ to finish? It will be simpler to `xargs -P4`. — KamilCuk, Mar 10 '23 at 11:08
@KamilCuk I want to wait for all 4 to finish and then submit more 4. — kamalika ray, Mar 10 '23 at 11:10

vmicrobio · Accepted Answer · 2023-03-10T11:33:51.960

2

In a bash script, you can try something like:

processes_max=4
counter=0

for f in input_files/*.txt
do
    python test.py -i $f -p parameter_1 -o output_files/$(basename $f) &  
    counter=$((counter+1))
    # if the counter equal to processes_max: wait for all processes to finish
    if [ $counter -eq $processes_max ]; then
        wait
        counter=0
    fi
done

wait

edited Mar 10 '23 at 11:33

answered Mar 10 '23 at 11:19

vmicrobio

331
1
2
13

1

Do not parse ls. Just `f in input_files/*.txt` – KamilCuk Mar 10 '23 at 11:32
1

Thanks a lot! I am trying to automate a pipeline. Will this also wait for all processes to finish before the next step starts? – kamalika ray Mar 10 '23 at 11:35
yes, with command 'wait' it will wait for all processes to finish before the next steps start – vmicrobio Mar 10 '23 at 11:38

score 1 · Answer 2 · answered Mar 10 '23 at 11:36

Wait needs to be placed after job creation.

for (( n = 0; n < 12; ++n )); do
    python test.py -i 'input_list[n]' -p parameter_1 -o "output_$n" &
    (( j = (j + 1) % 4 )) || wait
done

Optionally add another wait after the loop to wait for lesser than 4 processes.

score 0 · Answer 3 · answered Mar 10 '23 at 13:46

If you have GNU Parallel:

parallel -j4 python test.py -i {} -p parameter_1 -o {}.output ::: inputfiles*

If your system has 4 CPU threads, you can remove -j4.

It will start 4 jobs and when one job finishes, it will start the next.

This gives a better utilization of the CPU than if you wait for all 4 jobs to finish - especially if one of the jobs takes longer than the others.

Renaud Pacalet · Answer 4 · 2023-03-10T15:07:36.187

If you simply wanted to have at most 4 running jobs at a time GNU parallel would be your man:

printf '%s\n' "${input_list[@]}" |
  parallel -j4 'python test.py -i {} -p parameter_1 -o output_$(({#}-1))'

But as you want to wait until the four finish before you start the next batch (less efficient) you could group your 4 jobs in a single bash function and then use parallel with -j1 -N4:

foo() {
  ((n=4*("$1"-1))); shift
  for ((i=0;i<4;i++)); do
    python test.py -i "$1" -p parameter_1 -o output_"$((n+i))" & shift
  done
  wait
}
export -f foo
printf '%s\n' "${input_list[@]}" | parallel -j1 -N4 foo {#} {1} {2} {3} {4}

Run bash command in parallel for N-processes in batch

4 Answers4