Parallel processes in bash, kill with Ctrl C

Question

Just made my first bash script. It spawns a http-server process in parallel for each folder in a certain directory.

Here is what I have:

PORT=8001
cd websitesfolder
for d in */ ; do
  http-server $d -p $PORT &
  let "PORT++" 
done

This does spawn the http-servers, but when I exit with Ctrl C, the processes are not killed.

I implemented this without a loop first, and adding && fg I was able to kill the processes using Ctrl+C

http-server dir1 -p 8001 & http-server dir2 -p 8002 && fg

Is something similar possible with my looped solution?

edit: system is macOS

Best practice is not to do this at all, but to -- for example -- have a separate systemd service per instance. — Charles Duffy, Dec 10 '18 at 20:29
(...since it's been clarified that you're on MacOS, that would be launchd, not systemd). — Charles Duffy, Dec 10 '18 at 23:04
xargs might be a very good solution too :) https://stackoverflow.com/questions/28357997/running-programs-in-parallel-using-xargs — kalou.net, Dec 11 '18 at 07:44

Charles Duffy · Accepted Answer · 2018-12-10T20:57:25.620

4

A simple approach -- albeit not completely reliable, should a process die and have something else be assigned its PID number -- is simply to maintain an array of PIDs:

#!/usr/bin/env bash

pids=( )    
port=8001

graceful_shutdown() {
  (( ${#pids[@]} )) && kill "${pids[@]}"
}

cd websitesfolder || exit
for d in */ ; do
  http-server "$d" -p "$port" & pids+=( "$!" )
  (( ++port ))
done

trap graceful_shutdown EXIT
for pid in "${pids[@]}"; do
  wait "$pid" || (( retval |= $? ))
done
exit "$retval"

A better approach is to have a lockfile per port, or to use fuser -k to kill the processes that actually hold the ports open directly, rather than assuming that you know the PIDs:

#!/usr/bin/env bash

graceful_shutdown() {
  fuser -k "$lockdir"/*
}

lockdir=/var/run/mydir # this needs to exist
port=8001
pids=( )

cd websitesfolder || exit

for d in */; do
  (exec 3>"$lockdir/$port" &&
   flock -x 3 &&
   exec http-server "$d" -p "$port") & pids+=( "$!" )
done

trap graceful_shutdown EXIT
for pid in "${pids[@]}"; do
  wait "$pid" || (( retval |= $? ))
done
exit "$retval"

edited Dec 10 '18 at 20:57

answered Dec 10 '18 at 20:30

Charles Duffy

280,126
43
390
441

Thanks, can you elaborate on your comment why this is not best practice and how a systemd service would be better? – Roy Prins Dec 10 '18 at 20:31
1

So, you can configure a service like http-server@.service, where when you "systemctl start http-server@8001", it substitutes the instance name 8001 into the configured command. That way you can individually query each instance, configure restart behavior for each instance, command shutdown or start-on-boot behavior for each instance, ask for logs for any individual instance, etc. – Charles Duffy Dec 10 '18 at 20:38
Your shell script here doesn't know when the http-server instance running on port 8005 dies -- a supervision system (of which systemd is just one -- runit, supervisord, upstart, launchd, &c. are among the many others available) watching that service will be `wait()`ing for it continually, and will be notified / woken up the moment that happens. Having a system-wide name also means you get locking against multiple instances being started "for free". – Charles Duffy Dec 10 '18 at 20:39
I am accepting your answer as a further study case for myself, since I lack the basic skills to properly verify it. The first code sample simply completes without return value and for the second I lack `flock` on OSX. Not saying there is anything wrong, I simply lack the judgement. Thanks a lot for the pointers, hopefully I will be able to make more of it soon. – Roy Prins Dec 10 '18 at 20:48
Ahh, right -- we *do* need to actually `wait` for the processes. Amending appropriately. :) – Charles Duffy Dec 10 '18 at 20:56

Parallel processes in bash, kill with Ctrl C

1 Answers1