Node module: Don't return until all async requests have finished

Question

I'm new to node and am having trouble understanding node's async behavior. I know this is a very frequently addressed question on SO, but I simply can't understand how to get any of the solutions I've read to work in my context.

I'm writing this module which I want to return an object containing various data.

var myModule = (function () {
    var file,
    fileArray,
    items = [],

    getBlock = function (fileArray) {
        //get the data from the file that I want, return object
        return block;
    },

    parseBlock = function (block) {
        //[find various items in the block, put them into an "myItems" object, then
        //take the items and do a look up against a web api as below]...

        for (var i = 0, l = myItems.length; i < l; i ++) {
           (function (i) {
                needle.post(MY_URL, qstring, function(err, resp, body){
                    if (!err && resp.statusCode === 200){
                        myItems[i].info = body;
                    if (i === (myItems.length -1)) {
                        return myItems;
                        }
                    }
                });
            })(i);
        }
    },

    getSomeOtherData = function (fileArray) {
        //parse some other data from the file
    }

    return {
        setFile: function (file) {
            fileArray = fs.readFileSync(file).toString().split('\n');
        },

        render: function () {
            var results = [];
            results.someOtherData = getsomeOtherData();
            var d = getBlock();
            results.items = parseBlock(d);
            return results;
        }
    }
})();

When I call this module using:

myModule.setFile('myFile.txt');
var res = myModule.render();

the variable res has the values from the someOtherData property, but not the items property. I understand that my long-running http request has not completed and that node just zooms ahead and finishes executing, but that's not what I want. I looked at a bunch of SO questions on this, and looked at using Q or queue-async, but with no success.

How do I get this module to return no data until all requests have completed? Or is that even possible in node? Is there a better way to design this to achieve my goal?

`needle.post` is an async call, the callback is the 3rd parameter `function(err, resp, body)`. the solution is to be able to pass a callback into `myModule.render` which is invoked within the `needle.post` callback upon completion (for both failure or success, illustrating as such) — zamnuts, Dec 20 '13 at 00:27

score 0 · Accepted Answer · edited May 23 '17 at 12:03

The problem in your example is your calling getBlock() but you have declared your function as getBlockData(). So you will not get a result. Try changing it to both the same.

Presuming that you have them both the same, your next problem is that your processing data from a file, so I presume that your reading the contents of the file and then parsing it.

If this is the case then there are sync reads that you can use to force sync, however I wouldn't recommend this.

You really want to structure your program based on events. Your thinking in the paradigm of 'call a function, when it returns continue'. You need to be thinking more along the lines of 'call a process and add a listener, the listener then does reply handling'.

This works very well for comms. You receive a request. You need to reply based on contents of file. So you start the read process with two possible results. It calls the completed function or the error function. Both would then call the reply function to process how to handle a reply for the request.

It's important not to block as you will be blocking the thread via which all processes are handled.

Hope that helps, if not add some comments and I will try and elaborate.

Have a look at this answer to another question to see a good example of processing a file using the standard listeners. All async calls have a listener concept for what can happen. All you need to do is pass a function name (or anon if you prefer) to them when you call them.

A quick example (based on node.js stream.Readable API:

fs.createReadStream(filename, {
  'flags': 'r'
}).addListener( "data", function(chunk) {
  // do your processing logic
}).addListener( "end", function(chunk) {
  // do your end logic
  response(...);
}).addListener( "error", function(chunk) {
  // do your error logic
  response(...);
}).addListener( "close",function() {
  // do your close logic
});

function response(info) {
}

Thanks, I mistyped the function names when composing the question. I think I'm having problems with how to implement listeners in this module. Could you possibly point me to an example? — Judson, Dec 20 '13 at 00:43
No worries, thought it may have been a typo. I've added an extra paragraph with a link to an answer for a different question that has an example of processing a file. All you need to add is a function to generate the 'reply' and then call it from inside the appropriate listener function. — Metalskin, Dec 20 '13 at 01:04
Accepted this as the answer because it forced me to think in terms of callbacks and events. I used [node-google](https://github.com/jprichardson/node-google/blob/master/lib/google.js) as a model and also read deeply through [Callback Hell](http://callbackhell.com/). I ended up completely rewriting the code. — Judson, Dec 20 '13 at 12:17

Node module: Don't return until all async requests have finished

1 Answers1