How to wait for parallelized bash command that itself runs other tasks in threads? (doxygen & dot)

Question

I'm on Mac OS X and use the following bash script to run doxygen on around 10 source code libraries.

Problem is, doxygen runs the most time-consuming task "dot" (creates class diagrams) on separate threads by itself. This causes the wait command to not wait for the dot threads to finish. It effectively returns me to the command prompt right away (creating just the docs with doxygen is fast) while dot is still generating class diagrams in the background for another minute or so.

#!/bin/bash

DOXY="/Applications/Doxygen.app/Contents/Resources/doxygen"

#run doxygen on all configs
for config in doxygen*.config
do
    $DOXY $config &
done
wait

I have tried enforcing dot to run on a single thread (DOT_NUM_THREADS=1) but that didn't help. I tried putting the wait command in the loop, but that only has doxygen take as long as it does without parallelizing, and still not wait for dot to finish.

I need to wait for the dot command to generate all class diagrams, or else I will be packaging/uploading incomplete docs in the next step.

How can I parallelize doxygen and have it wait for all threaded dot commands to finish?

Alternatively, is there a way for me to determine in the bash script if all dot commands have ended (I do not know what processes/threads doxygen actually spawns nor how to wait for them)?

UPDATE: I tried using xargs which properly waits for all dot threads to finish, but isn't any faster:

find . -name "doxygen*.config" | xargs -n 1 -P 8 doxygen

Problem here is I can't set 'n' (number) higher than 1 because then doxygen won't create all the docsets because it ignores the additional arguments.

score 3 · Answer 1 · edited May 23 '17 at 11:50

3

Have a look here https://stackoverflow.com/a/8247993/784672, where I describe a way to run doxygen in parallel using tag files and the GNU parallel utility.

edited May 23 '17 at 11:50

Community

1
1

answered Mar 16 '13 at 10:26

doxygen

14,341
2
43
37

That's a good approach, but doesn't seem to apply in my situation. 99% of the time is spent creating the images with dot in my case, doxygen itself is almost instantly finished. – CodeSmile Mar 16 '13 at 16:00

score 0 · Answer 2 · answered Mar 16 '13 at 09:35

I found a solution. But first, I was a fool to believe I can speed up doxygen. I already mentioned that doxygen is threading the most time-consuming task (dot). Therefore: duh! :)

Anyway, what worked for me was to create another bash script with the doxygen command:

#!/bin/bash
DOXY="/Applications/Doxygen.app/Contents/Resources/doxygen"
echo Making $1 …
$DOXY $1

And then the original bash script can parallelize the task, and correctly waits for dot to finish:

#!/bin/bash

#run doxygen on all configs
for config in doxygen*.config
do
    ./make-doc.sh $config &
done
wait

score 0 · Answer 3 · answered Mar 16 '13 at 13:51

I think another solution (not tested, as I don't have doxygen installed), is to just run doxygen as a foreground task in a subshell that is backgrounded. This is pretty much the same as the solution you found, but without the need to create a second script:

for config in doxygen*.config; do
    ( $DOXY $config ) &
done
wait

How to wait for parallelized bash command that itself runs other tasks in threads? (doxygen & dot)

3 Answers3