Bash wait does not work

Question

Part of my code:

for i in "${r_arr[@]}"
do
    ${parallel_dir}parallel -j${cap} --semaphore --semaphorename a2_bootstrap_semaphore_${rnd} a2_exec ${i} ${temp_zmat_folder_path}${temp_zmat_folder}/ ${outs_folder}
done

wait
elapsed_time=$(($SECONDS - $start_time))

rm -rf "${temp_zmat_folder_path}${temp_zmat_folder}"
echo "Cleanup done."
echo "`date +%d-%m-%y` `date +%T`: All processing finished in ${elapsed_time} seconds!"

As you can see I'm starting a number of tasks using GNU Parallel. The problem is that cleanup (rm) is executed before last of the parallel tasks is even started. How to prevent such behaviour?

I don't see an asynchronous process to wait for in your script. No `&` anywhere... — Kusalananda, Jul 28 '16 at 18:22
`wait` only waits for direct children of the shell; it cannot be used to `wait` on processes started by other processes (such as `parallel`). — chepner, Jul 28 '16 at 18:36

score 4 · Accepted Answer · edited Sep 13 '20 at 15:08

4

As per their example, you should use this command to wait for all the started jobs to complete

parallel --semaphore --wait

Example (From man page): The command sem is an alias for parallel --semaphore.

for i in *.log ; do
    echo $i
    sem -j+0 gzip $i ";" echo done
done
sem --wait

edited Sep 13 '20 at 15:08

Gorka

1,971
1
13
28

answered Jul 28 '16 at 18:31

Sam Daniel

1,800
12
22

So the `wait` part has to be called from the `sem` itself. I missed that in docs, thanks. – alex Jul 28 '16 at 19:19

score 0 · Answer 2 · answered Jul 29 '16 at 06:23

Sam Daniels answer is entirely correct (except you need to wait for the --semaphorename that you use), but sem is really slow. Consider using a function instead:

doit() {
  i="$1"
  a2_exec ${i} ${temp_zmat_folder_path}${temp_zmat_folder}/ ${outs_folder}
}
export -f doit
export temp_zmat_folder_path
export temp_zmat_folder
export outs_folder
${parallel_dir}parallel -j${cap} doit ::: "${r_arr[@]}"

Bash wait does not work

2 Answers2