How to make an array of Deferred objects

Question

Am new to Deferreds and Promises.

Here is my [simplified] code, which is defined within a JavaScript object:

myFunction: function(d, cb)
{
    return $.ajax('/myURL', {
        contentType:    'application/json',
        data:           d,
        dataType:       'json',
        type:           'POST'
    }).then(cb, cb);
},

flush: function(myArray)
{
    return myFunction(myArray, myCallback);
}

The above works fine. I can call flush(someArray), and some time later I get the result of the ajax request.

QUESTION:

I want to modify the flush function so that it first breaks the array into chunks (i.e. smaller arrays), and then calls myFunction on each of those chunks. It must then return, in one go, the aggregated data (preferably in an array) from each of the ajax calls that were made.

I am starting to modify flush() along the following lines, but I know it's not quite right. Please can someone complete/fill in the gaps for me, or suggest a re-structuring that would work well?

Thanks.

flush: function(myArray)
{
    var chunk = 2;
    var i, a;
    var j = myArray.length;

    var myArrayChunks = [];

    for (i=0; i<j; i+=chunk)
    {
        a = myArray.slice(i, i+chunk);
        myArrayChunks.push(a);
    }

    var myDeferreds = [];

    for (i=0; i<myArrayChunks.length; i++)
    {
        // Here, I need to create a deferred object that will run: myFunction(myArrayChunks[i], myCallback)
        // How do I do that?

        var f = // The deferred object that will run: myFunction(myArrayChunks[i], myCallback)

        myDeferreds.push(f);
    }   

    return $.when.apply($, myDeferreds).then(function(){

        // Here, I need to get the aggregated data that is returned by each of the deferreds. How do I do that?

        console.log("FLUSH COMPLETE!");
    }); 
}

Answer 1

The async library I've pasted below allows you to run a series of async/deferred requests, and passes on the results of each async function to a final callback, which aggregates a collection of the results.

In particular, check out the parallel method, which will execute all of your async requests simultaneously, but there is no guarantee which order they will run in. If you are concerned about which order to execute your async requests, check out the seriesand eachSeries methods.

parallel:

https://github.com/caolan/async#parallel

series / eachSeries:

https://github.com/caolan/async#seriestasks-callback

Both methods aggregate your results into a final results object, which contains all of the results passed on from each async call you make.

NOTE, to use jQuery's deferred functionality, you would need to call .resolve() in the "final" callback of the async.parallel or async.each or async.eachSeries methods

Here's an example of the parallel method:

async.parallel([
    function(callback){
        // some request
        $.ajax(/*details*/, function(data) {
            callback(null, data);
        });
    },
    function(callback){
        // some request
        $.ajax(/*details*/, function(data) {
            callback(null, data);
        });
    }
],

// "final" callback, invoked after all above functions have
// called their respective callback() functions
function(err, results){
    if(err) {
        // handle error
    } else {
        // results contains aggregated results from all 
        // async calls (2nd parameter in callback(errorParam, resultsParam)
        console.log('all async methods finished!', results);
    }
});

Here's a way to pass in an array and make async methods with each array element. NOTE that every async call within the async.each method must call callback() when the async request is resolved, or callback(err) in your async error method if there an error. If you pass in an array of N elements to the async.each method, the final callback will be invoked when all N async resolve callback() methods have been invoked.

async.each(array, function(element, callback) {
  
  $.ajax(/* details */, {data: element}, function(data) {
      // call `callback` when you're finished up
      callback();
  });

}, 
// "final" callback, invoked after each async call is resolved and
// invokes the callback() function
function(err){
    if( err ) {
      // handle errors
    } else {
      console.log('All async methods flushed!');
    }
});

I love this library, and once you start using it it'll change your life :]. Best of luck!

Answer 2

Since you already have a promise returned from your ajax function, I'd suggest you use promises instead of plain callbacks. Here's a way to do that:

myFunction: function(d) {
    return $.ajax('/myURL', {
        contentType:    'application/json',
        data:           d,
        dataType:       'json',
        type:           'POST'
    });
},

flush: function(myArray, chunkSize) {
    chunkSize = chunkSize || 2;
    var index = 0;
    var results = [];
    var self = this;

    return jQuery.Deferred(function(def) {
        function next() {
            var start = index;
            var arrayChunk, promises = [];
            index += chunkSize;
            if (index < myArray.length) {
                arrayChunk = myArray.slice(start, chunkSize);
                // create chunkSize array of promises
                arrayChunk.forEach(function(item) {
                    promises.push(self.myFunction(item));
                });
                $.when.apply($, promises).then(function() {
                    // results are in arguments[0][0], arguments[1][0], etc...
                    for (var i = 0; i < arguments.length; i++) {
                        results.push(arguments[i][0]);
                    }
                    // next iteration
                    next();
                }, def.reject)
            } else {
                def.resolve(results);
            }
        }

        // start first iteration
        next();

    }).promise();

}

obj.flush(myArray).then(function(results) {
    // array of results here
}, function(jqXHR, textStatus, errorThrown) {
    // error here
});

Answer 3

Here's another way to do it by creating a version of $.ajax() which I call $.ajaxChunk() that takes an array of data and does the chunking for you.

// Send ajax calls in chunks from an array with no more than X in flight at the same time
// Pass in array of data where each item in dataArray is sent separately
//    in an ajax call
// Pass settings.chunkSize to specify the chunk size, defaults to 2 if not present
// Returns a promise
//   The resolved value of promise is an array of results
//   The rejected value of the promise is whatever jQuery result failed first
$.ajaxChunk = function(dataArray, url, settings) {
    settings = settings || {};
    var chunkSize = settings.chunkSize || 2;
    var index = 0;
    var results = [];
    return jQuery.Deferred(function(def) {

        function next() {
            var start = index;
            var arrayChunk, promises = [];
            index += chunkSize;
            if (index < myArray.length) {
                arrayChunk = myArray.slice(start, chunkSize);
                // create chunkSize array of promises
                arrayChunk.forEach(function(item) {
                    // make unique copy of settings object for each ajax call
                    var localSettings = $.extend({}, settings);
                    localSettings.data = item;
                    promises.push($.ajax(url, localSettings));
                });
                $.when.apply($, promises).then(function() {
                    // results are in arguments[0][0], arguments[1][0], etc...
                    for (var i = 0; i < arguments.length; i++) {
                        results.push(arguments[i][0]);
                    }
                    next();
                }, def.reject)
            } else {
                def.resolve(results);
            }
        }

        // start first iteration
        next();

    }).promise();

}

And, sample usage:

$.ajaxChunk(arrayOfData, '/myURL', {
    contentType:    'application/json',
    dataType:       'json',
    type:           'POST',
    chunksize:      2
}).then(function(results) {
    // array of results here
}, function(jqXHR, textStatus, errorThrown) {
    // error here
})

---

If the real requirement here is that you don't have more than X ajax calls in process at the same time, then there's a more efficient and faster (end-to-end time) way to do than chunking. Instead, you keep track of exactly how many ajax calls in "in flight" at any time and as soon as one finishes, you start the next one. This is a bit more efficient than chunking where you send the whole chunk, then wait for the whole chunk to finish. I've written a jQuery helper that implements this:

$.ajaxAll = function(dataArray, url, settings, maxInFlight) {
    maxInFlight = maxInFlight || 1;
    var results = new Array(dataArray.length);
    settings = settings || {};
    var index = 0;
    var inFlight = 0;
    return jQuery.Deferred(function(def) {

        function runMore() {
            while (inFlight < maxInFlight && index < dataArray.length) {
                (function(i) {
                    var localSettings = $.extend({}, settings);
                    localSettings.data = dataArray[index++];
                    ++inFlight;
                    $.ajax(url, localSettings).then(function(data, textStatus, jqXHR) {
                        --inFlight;
                        results[i] = data;
                        runMore();
                    }, def.reject);
                })(index);
            }
            // if we are all done here
            if (inFlight === 0 && index >= dataArray.length) {
                def.resolve(results);
            }
        }

        // start first iteration
        runMore();

    }).promise();
}

Note: If you pass 1 for the maxInFlight argument, then this runs the ajax calls in series one after the other. Results are always returned in order.

And, sample usage:

$.ajaxAll(arrayOfData, '/myURL', {
    contentType:    'application/json',
    dataType:       'json',
    type:           'POST'
}, 2).then(function(results) {
    // array of results here
}, function(jqXHR, textStatus, errorThrown) {
    // error here
})

Answer 4

Thanks to all for great advice.

I used a combination of the suggested techniques in my solution.

The key thing was to make an array of promises, and push onto it the required calls (each with its own array chunk passed as a parameter) to the function that makes the ajax request. One thing I hadn't previously realised is that this calls the ajaxCall() function at that very moment, and that's ok because it returns a promise that is pushed onto the array.

After this, the 'when.apply' line does the trick in waiting until all the ajax promises are fulfilled. The arguments of the 'then' function are used to collate all the results required (obviously, the exact mechanism for that depends on the format of your returned arguments). The results are then sent to theResultsHandler(), which takes the place of the original callback in the code I first posted in my question.

Hope this is useful to other Promise-novices!

The ajax-calling function is:

ajaxCall: function(d) {
    return $.ajax('/myURL', {
    contentType:    'application/json',
    data:           d,
    dataType:       'json',
    type:           'POST'
    });
},

And inside the flush() function...

    var promises = [];
    var i, j;

    for (i=0; i<batchChunks.length; i++)
    {
        promises.push(self.ajaxCall(batchChunks[i]));
    }   

    var results = [];

    return $.when.apply($, promises).then(function(){

        console.log("arguments = " + JSON.stringify(arguments));

        for (i = 0; i < arguments.length; i++)
        {
            for (j = 0; j < arguments[i][0].length; j++)
            {
                results.push(arguments[i][0][j]);
            }
        }

        return self.theResultsHandler(results);
    });