async.parallel for large number of functions

Question

I have a function that I want to call 1 million times. The function makes two calls to the database (first a SELECT and then an UPDATE). My current approach is to store these functions in an array and then call async.parallel on this array.

I am afraid that it would either result in ENOMEM or something.

Any better approach here ?

Try using `async.eachLimit` to throttle the calls. Otherwise I think you'll break the stack. — Razvan, Oct 30 '15 at 18:28
you don't need a separate function in each array index, referencing the same function over and over would do the same. parallel will work, if you limit the number of concurrent calls to a reasonable number. **however**, the *right* solution would be to eliminate the need to send 2million different queries. — Kevin B, Oct 30 '15 at 18:29

score 4 · Accepted Answer · answered Oct 30 '15 at 19:18

You can do a queue with a generator:

var totalTasks = 1000000;
var nTasks = 0;

// Generate data to send to query 
var getData = (function() {
    var i = 0;

    return function() {
        i++;
        return {
            number: i
        };
    };

})();

// Create the task running each time
var task = function(data, done) {
    nTasks++;
    console.log("Make task", nTasks);
    doQuery(data, done);
};

// Create queue with a limit concurrency
var queue = async.queue(task, 10); // <- parallels queries*

// The callback execute each task was execute
var cb = function(err) {
    if (err) throw err;

    // Add new tasks to queue if is neccesary    
    if (nTasks < totalTasks && queue.length() < queue.concurrency) {
        queue.push(getData(), cb);
    }

};

var i;

// Add the first x tasks to queue
for (i = 0; i < queue.concurrency; i++) {
    queue.push(getData(), cb);
}

You need to consider the size of the connections pool, or the number of max current proccess of MySQL.

Thanks, i ended up using a slightly different approach taking ideas from this answer. Used --harmony flag in node 0.11 and wrote a generator function. — pranavk, Nov 01 '15 at 14:02

score 0 · Answer 2 · answered Oct 30 '15 at 18:32

0

if you want to process each row , you could use Mysql stream , so you can apply whatever you want to each row (update in your case);

answered Oct 30 '15 at 18:32

cshion

1,173
1
10
19

async.parallel for large number of functions

2 Answers2