I'm currently using Pheanstalk (beanstalkd) to process a number of jobs using multiple workers. I'm wondering if there is a common way to determine if all jobs are finished, as the workers are run asynchronously.
I've thought of adding a "finished" flag to the last job issued, and use workers to check for that flag. The problem is the last job isn't always the last one finished by workers. Workers can get hung up, crash etc.
If you want to know when they're all finished, you'll need each job to flag its completion, and wait until all flags are set to complete. As you've pointed out, there's no guarantee on order of completion for a set of jobs across multiple workers.
Could that work be queued as a single job, instead of multiple, so that it's a simple matter of waiting for that single job to finish?
Perhaps you should also think about whether you need to know when these jobs are finished. e.g. for image processing, one can display a placeholder image until the actual image is ready. If you really need to wait for them to all finish, then maybe they're not best suited to an asynchronous work queue.
To get a better answer, you might need to elaborate on your question, or provide a more concrete example.
Cheers! Paul