I have a multi-process application that works like so...
There is a parent process. The parent process queries a database to find work, then forks children to process that work. The children communicate back to the parent via System V message queues to indicate they're done with their work. When the parent process picks up that message, it updates the database to indicate that the work is complete.
This works okay but I'm struggling with handling the parent process being killed.
What happens is the parent receives a SIGINT(from CTRL-C), and then sends SIGKILLs to each of the children. If a child is currently blocking on a Sys V message queue write when it receives that signal, the write is "interrupted" by the signal and the blocking canceled and the parent never learns that the child's work was done, and the database never gets updated.
That means that the next time I run the script, it will re-run any work that was blocking on the System V queue write.
I don't have a good idea for a solution for this yet. Ideally I would like to be able to force the queue write to remain blocking even when it receives that SIGKILL but I don't think such a thing is possible.