handle multiple request to node js api

Question

I was requested to create a simple rest api with node js and make a script that creates 10000 elements in the database through api calls. I created the server with the Hapi framework. If I send a single or 100 'PUT' requests to the API it creates a new element without problem but if I try to make 1000 requests or more it wont be able to create all of them or anything at all. I would like to know what may be the problem, if I'm not doing the script correctly or is the server the problem. So far I've received 2 errors:

{ [Error: connect ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'connect' }

and (libuv) kqueue(): Too many open files in system

The first one trying to call the api 1000 times and the second one trying with 10000 times

The code of the server is the following

var Hapi = require('hapi');
var server = new Hapi.Server();
var joi = require("joi");

var dbOpts = {
    "url" : "mongodb://localhost:27017/songsDB",
    "settings" : {
        "db" : {
            "native_parser" : false
        }
    }
};

server.register({
    register: require('hapi-mongodb'),
    options: dbOpts
},function (err) {
    if (err) {
    console.error(err);
    throw err;
}
});


server.connection({
    host: 'localhost',
    port: 8080
});


server.route({
    method: 'POST',
    path: '/songs',
    config: {
        handler: function (request, reply) {
            var newSong = {
                name: request.payload.name,
                album: request.payload.album,
                artist: request.payload.artist
            };

            var db = request.server.plugins['hapi-mongodb'].db;
            db.collection('songs').insert(newSong, {w:1}, function (err, doc){
                if (err){
                    return reply(Hapi.error.internal('Internal MongoDB error', err));
                }else{
                    reply(doc);
                }
            });
        },
        validate:{
            payload: {
                name: joi.string().required(),
                album: joi.string().required(),
                artist: joi.string().required()
            }
        }
    }
});

server.start(function () {
    console.log('Server running at:', server.info.uri);
});

The code for the request is the following

var unirest = require('unirest');

for(var i = 1; i<=10000; i++){
unirest.post('http://localhost:8080/songs')
.header('Accept', 'application/json')
.send({ "name": "song"+i, "artist": "artist"+i, "album":"album"+i})
.end(function (response) {
    console.log(response.body);
});
}

What OS are you running this on? – RobertoNovelo Apr 14 '15 at 00:23 — RobertoNovelo, Apr 14 '15 at 00:23

score 0 · Answer 1 · answered Apr 14 '15 at 00:34

0

If running under OSX, open terminal and then try using:

sudo launchctl limit maxfiles 1000000 1000000

and then try again.

answered Apr 14 '15 at 00:34

RobertoNovelo

3,347
3
21
32

score 0 · Answer 2 · answered Apr 14 '15 at 02:29

For "too many open files in system", looks like you reach the limit of your system. If you are using Linux, you can do a ulimit -a to display all settings. There is one which may limit your number of open files. open files (-n) 1024

score 0 · Answer 3 · edited May 23 '17 at 12:14

Assuming you are on a Mac or Linux, you need to increase the maximum number of open files allowed by the system.

If you insert this into the terminal it will show you what your settings are:

lsof | wc -l

You will see that your 'open files' setting is likely smaller than the number you are trying to work with.

To change this setting, use the following command:

ulimit -n #####

where ##### is some arbitrary number (but higher than what you have).

If you are on a Windows machine, the answer is slightly more complicated. It seems Windows has a per-process limit, which can be modified (though it doesn't sound easy). Look here for some more details: Windows equivalent of ulimit -n

When I ran the code, the first 11 POSTs would throw errors. Apparently it is because the script begins sending them before the mongodb connection is active. All I did was added a brief timeout to the POSTing to give mongodb a chance to start breathing. When I did that it worked fine. 10000 records upon completion.

All I changed was this:

setTimeout(function () {
  for(var i = 1; i<=10000; i++){
    unirest.post('http://localhost:8080/songs')
        .header('Accept', 'application/json')
        .send({ "name": "song"+i, "artist": "artist"+i, "album":"album"+i})
        .end(function (response) {
            //console.log(response.body);
        });
  }
}, 5000);

Thank you for the information, now the open files issue is gone. Although when I try to make the 10000 request many of them cannot be processed and no element is added to the database. Do you know what could be the problem here? — George Hdez, Apr 14 '15 at 12:30
Just curious, but is there a reason you are using node to send Ajax calls to itself? — tpie, Apr 14 '15 at 12:41
I am starting to learn node so I found it convenient to do both the server and the requests script with node. — George Hdez, Apr 14 '15 at 13:59

handle multiple request to node js api

3 Answers3