2

I am using the request-promise module in Node.js to produce lots of parallel requests (>1000), and I am getting the following error after about half of them (500) have completed:

Error: getaddrinfo ENOTFOUND

I saw this github issue: https://github.com/nodejs/node-v0.x-archive/issues/5545 which appears to be my problem, but they never came up with a satisfactory solution. Also, that was for node versions v0.x, while this is occurring in version 5.7.0.

Just for clarity, my requests are approximately the following:

var urls = [];
for(var i=0; i<1000; i++) {
    //create url
    urls.push(url);
}
return Promise.all(urls.map(createAPIRequestPromise));

where createAPIRequestPromise is essentially:

var rp = require('request-promise');
function createAPIRequestPromise(url) {
    var options = {
        headers: {
            'APIKey': apiKey
        },
        json: true,
        url: url
    };
    return rp(options);
}

Edit:

Use of promises: if we let the the first block of code be a function called firerequests then it looks something like:

firerequests().then(function(bodies){
    bodies.forEach(function(body){
        console.log(body.valueX);
    });
}).catch(function(err){
    console.log(err);
});
user6517436
  • 111
  • 1
  • 5
  • 1
    I'd suggest not trying to launch 1000 requests at a time. You are probably running out of some sort of system resource. I'd suggest only running 10-20 requests at a time. If that works well, you can experiment with how much higher you can safely raise the number and whether it actually helps your performance or not. FYI, you can use something like Bluebird's `Promise.map()` and its concurrency value to automatically manage how many requests are in-flight at the same time while still collecting all the values for you via promises. – jfriend00 Jun 30 '16 at 06:03
  • can you also add the part of the code where you use the promise – ozata Jun 30 '16 at 06:04
  • @ozata I have added a little bit of code for how I deal with the promise – user6517436 Jun 30 '16 at 20:20
  • @jfriend00 I forgot to mention this in the post, but I have actually tried using Bluebird's `Promise.map()` for this and I got the same result when concurrency was 10 (I just ran it and it made it a little further 750 out of 1000, but failed with the same error, anyways) – user6517436 Jun 30 '16 at 20:25
  • can you try returning a `new Promise` instead of `Promise` only – ozata Jul 01 '16 at 06:45
  • As far as I can tell your issue is not related to request-promise and not even related to request which is used under the hood. Request uses the http or https module of node to actually send the requests. Maybe it is a good idea you set up a test script that uses the http module directly and then figure out how to make it work. – analog-nico Jul 16 '16 at 07:27

0 Answers0