Multiple API connections through Nodejs

Question

I want to connect to theMovieDb api, and fetch a large number of movies. The api limit is 40 calls every 10 seconds and 20 connections per IP.

How do I do this in node.js? Right now I have one http request to the api at a time, but anything more than 40 calls results in errors.

score 1 · Accepted Answer · answered Dec 19 '15 at 12:53

1

40 requests per 10 second, 20 concurrent connections, so in "one thread" we'll have to wait approximately 5 seconds.

There's an awesome library for all async stuff called async.

var loadOne = function (id, callback) {
   // Do your stuff
   // tell it to wait
   setTimeout(callback, 5000);
} 


async.mapLimit(arrayOfIds, 20, loadOne, function(err, results){
    // results is now an array of stats for each file
});

answered Dec 19 '15 at 12:53

Alexander Mikhalchenko

4,525
3
32
56

I might be wrong here. I know async and using that I can send multiple requests, but I dont get the concurrent connection part ? How is it done in node.js – shiv Dec 19 '15 at 13:09
@shiv Do you mean how it's done on the language-level? – Alexander Mikhalchenko Dec 19 '15 at 13:11
@shiv In the code above we just limit the number of concurrent actions using ``async`` – Alexander Mikhalchenko Dec 19 '15 at 13:11
If you're wondering in what way nodejs is asynchronous, cosider reading [this article](https://nodesource.com/blog/understanding-the-nodejs-event-loop/), it'll help you with that ;) – Alexander Mikhalchenko Dec 19 '15 at 13:18
@shiv Basicaly you use the same request in ``loadOne`` or any other http library you want. The whole point is that using ``mapLimit`` you can limit how many ``loadOne`` functions would be called parallel. – Alexander Mikhalchenko Dec 19 '15 at 13:20

Multiple API connections through Nodejs

1 Answers1