how to synchronize 2 async.waterfalls

Question

I have a set of read commands that I have to in sequence. Any fails, processing stops.

readCommands is an array of read functions...

async.waterfall(readCommands, function(err) {
    if (err) {
        console.log(err);
        res.send(500, { error : err.toString() });
        return;
    }
    else {
        console.log('we determined the state to which we have to rollback to');
    }
});

at this point, I know what I started with. Now I want to do the write commands

async.waterfall(writeCommands, function(err) {
    if (err) {
        console.log(err);
        res.send(500, { error : err.toString() });
        return;
    }
    else {
        console.log('we succeeded with all the write commands...');
    }
});

the arrays readCommands and writeCommands entries are radically different so very hard to combine them. But I DO for the first waterfall to finish before I go to the next one. How do I make a "waterfall" out of the existing two?

Is the data you gathered from the `readCommands` used as input to the `writeCommands`? — Nate, Dec 18 '13 at 00:52
Promises! That is what you want. http://promises-aplus.github.io/promises-spec/ — Sukima, Dec 18 '13 at 00:53
@Sukima, I thought about making each waterfall into a promise, is that what you mean? I really do not want any processing in this function until the reads are completed first and then the write and only then do more synchronous calls. That is why each waterfall could not have been replaced with promises. For one thing, the exception handling on the first waterfall is radically different that the second — reza, Dec 18 '13 at 16:11
In that case the easiest solution is what @jibsales recommends below: nested async methods. Although I’d probably use `series()` as the outer function instead of `waterfall()`. Although it can work either way, the `waterfall` method implies that the subsequent step will make use of the result of the previous step, which isn’t the case here. — Nate, Dec 18 '13 at 17:50
@Nate I interpreted "How do I make a "waterfall" out of the existing two" quite literally. You are right, however, to use `series` instead as there is no benefit to using `waterfall`. I've updated my answer. — srquinn, Dec 18 '13 at 19:32
@reza promises are design for just that case. You chain then and then serialize them. Something like this: https://gist.github.com/sukima/8034613 — Sukima, Dec 19 '13 at 04:57
@Sukima in your post, what does readCommands look like? an array of promises? or readCommands as I have shown ? an array of read functions to call . Your approach looks ultra simple, very elegant but I have to understand what readCommands look like — reza, Dec 20 '13 at 16:49
readCommands was a copy/paste from your code. In my example it simple returns a promise object. Obviously your readCommands would need to be refactored to return a promise. Do you have code of what readCommands does currently and I could try showing you what a refactored version would look like. — Sukima, Dec 21 '13 at 05:17

srquinn · Answer 1 · 2013-12-18T19:30:45.933

2

Sounds crazy, but you can actually nest these async methods:

async.series([
  function (done) {
    async.waterfall(readCommands, done);
  },
  function (done) {
    async.waterfall(writeCommands, done);
  }
], function (err) {
  // All done
});

edited Dec 18 '13 at 19:30

answered Dec 18 '13 at 00:57

srquinn

10,134
2
48
54

score 2 · Accepted Answer · answered Dec 18 '13 at 02:07

2

Just combine your two arrays, and they will run in sequence:

async.waterfall(readCommands.concat(writeCommands), function(err) {...}

answered Dec 18 '13 at 02:07

punund

4,321
3
34
45

This is working wonderfully, but now i have to pass a result variable. So each successfull call will need to have callback(null, anotherdataelement). When I make the call as stated above, I get Object is not a function. Any ideas how to get around that? – reza Dec 20 '13 at 18:47

how to synchronize 2 async.waterfalls

2 Answers2