12

I'm making a bunch calls to a database that contains a large amount of data on a Windows 7 64 bit OS. As the calls are queuing up I get the error (for ever HTTP call after the first error):

Error: connect ENOBUFS *omitted* - Local (undefined:undefined)

From my google searching I've learned that this error means that my buffer has grown too large and my system's memory can no longer handle the buffer's size.

But I don't really understand what this means. I'm using node.js to with an HTTPS library to handle my requests. When the requests are getting queued and the sockets are opening is the buffer's size allocated in RAM? What will allow the buffer to expand to a greater size? Is this simply a hardware limitation?

I've also read that some OS are able to handle the size of the buffer better than other OS's. Is this the case? If so which OS would be better suited for running a node script that needs to fetch a lot of data via HTTPS requests?

Here's how I'm doing my requests.

for (let j = 0; j < dataQueries; j++) {
 getData(function())
}

function getData(callback){
 axios.get(url, config)
   .then((res) => {
      // parse res 
      callback(parsedRes(res))
   }).catch(function(err) {
      console.log("Spooky problem alert! : " + err);
   })
}

I've omitted some code for brevity, but this is generally how I'm doing my requests. I have a for loop that for every iteration launches a GET request via axios.

I know there is an axios.all command that is used for storing the promise the axios.HTTPMethod gives you, but I saw no change in my code when I set it up to store promises and then iterate over the promises via axios.all

A. Werner
  • 385
  • 1
  • 2
  • 15
  • could we may see your code? i think the best solution would be some sort of qeue – Jonas Wilms Jun 21 '17 at 14:58
  • Also have a look at https://stackoverflow.com/questions/10603860/very-simple-node-js-client-throws-error-enobufs-after-many-http-requests – Jonas Wilms Jun 21 '17 at 14:59
  • @Jonasw added a brief snippet. Could you possibly link me to a queue example or give me the general idea to set it up? – A. Werner Jun 21 '17 at 15:15

4 Answers4

11

Thanks @Jonasw for your help, but there is a very simple solution to this problem. I used the small library throttled-queue to get the job done. (If you look at the source code it would be pretty easy to implement your own queue based on this package.

My code changed to:

const throttledQueue = require('throttled-queue')

let throttle = throttledQueue(15, 1000) // 15 times per second

for (let j = 0; j < dataQueries; j++) {\
 throttle(function(){
   getData(function(res){
     // do parsing
   })
 }

}

function getData(callback){
 axios.get(url, config)
   .then((res) => {
      // parse res 
      callback(parsedRes(res))
   }).catch(function(err) {
      console.log("Spooky problem alert! : " + err);
   })
}
A. Werner
  • 385
  • 1
  • 2
  • 15
  • 1
    Perfect, this was able to slow down the HTTP requests enough for node and my backend to handle...thank you! – cjones26 Aug 24 '17 at 16:54
3

In my case this got resolved by deleting the autogenerated zip files from my workspace, which got created every time I did cdk deploy. Turns out that my typescript compiler treated these files as source files and counted them into the tarball.

kingshuk
  • 472
  • 1
  • 4
  • 10
0

Youre starting a lot of data Queries at the same time. You could chain them up using a partly recursive function, so that theyre executed one after another:

(function proceedwith(j) {
   getData(function(){
     if(j<dataQueries-1) proceedwith(j+1); 
   });
})(0)
Jonas Wilms
  • 132,000
  • 20
  • 149
  • 151
0

Experienced the same issue when starting too many requests.

Tried throttled-queue, but wasn't working correctly.

system-sleep worked for me, effectively slowing down the rate at which the requests were made. Sleep is best used in synchronized code, to block before using sync/async code.

Example: (using sleep to limit the rate updateAddress() is called)

// Asynchronus call (what is important is that forEach is synchronous)
con.query(sql, function (err, result) {
    if (err) throw err;
    result.forEach(function(element) {
        sleep(120); // Blocking call sleep for 120ms
        updateAddress(element.address); // Another async call (network request)
    });
});
dalslandan200
  • 59
  • 6