How to divide workers and aggregate results?

Question

Given a payload submitted to a parentWorker:

1. I divide the work amongst otherWorkers and I add the taskId of the parentWorker as an additional property in the payload
2. each of the otherWorkers finish what they are responsible for

What I want is to know if 5 or 10 or 20 otherWorkers were queued/kicked off then when do they all finish? Because when all of them are finished, I want to start the next part of my workflow: nextWorker!

So the ideal pipeline is: parentWorker > X # of otherWorkers > everyone done? > nextWorker

How can I make this happen?

Please don't answer with a polling-based solution. I'm not looking for that.

I thought of using a cache:

1. where the parentWorker would set the total # of otherWorkers that will be created, like: cachekey_<parertTaskId>_workersCreated: 10
2. then the otherWorkers would decrement the # atomically by -1 after they finish and eventually the count will reach zero: cachekey_<parertTaskId>_workersCreated: 0 but who is supposed to act on that count?

a) If the idea is to let otherWorkers decrement it then check the value and to see if it is zero and kick off nextWorker ... that is flawed in a situation where:

cachekey_<parertTaskId>_workersCreated: 2
otherWorker9 sends -1
otherWorker10 sends -1
otherWorker9 checks and otherWorker10 checks
both get back 0 and both will kick off nextWorker! We only wanted one instance.

b) other bad ideas:

cachekey_<parertTaskId>_workersCreated: 2
otherWorker9 checks and otherWorker10 checks
neither one kicks off nextWorker because value!==1
otherWorker9 sends -1
otherWorker10 sends -1
its over and noone is left to act on cachekey_<parertTaskId>_workersCreated: 0

Answer 1

There's unfortunately no automated/very simple/built-in way to do this.

Regarding your idea to use a cache, if you use something like Redis, it's increment and decrement operations are atomic so you'd never get a case where both workers got back the same number. One worker and one worker only would get the zero back: http://redis.io/commands/decr