-1

I have a list of 125,000 + Id numbers.

I am making a request to an api to get more information for each one.

But my problem is that the api will stop giving me a response if I request more then 6 per second.

I need a way to control the speed of the requests.

Cœur
  • 37,241
  • 25
  • 195
  • 267
Bill
  • 4,614
  • 13
  • 77
  • 132
  • 1
    Hard to know without knowing the structure of your code, but how about using `setInterval(function() {}, 167)`? That should keep 6/s on average, but may go temporarily lower or higher due to processing delays. – Joachim Isaksson Jan 22 '16 at 08:29

2 Answers2

1

Just use a function called by setInterval to do the actual API querying ?

Simple example:

var ids = [ /* big array here */ ];

function queryAllIds(ids, callback) {
  var processedIds = [];
  var lastProcessedId = 0;
  var processedIdCount = 0;
  var intervalId;

  function queryApi() {
    var idToProcess = lastProcessedId++;
    doActualQuery(ids[idToProcess], function(result) {
      processedIds[idToProcess] = result;
      processedIdCount++;

      if (processedIdCount === ids.length) {
        nextTick(callback, processedIds);
      }
    });
  }

  if (intervalId && lastProcessedId === ids.length)
    clearInterval(intervalId);
  }

  intervalId = setInterval(queryApi, 1000/6);      
}

queryAllIds(ids, function (processedIds) { 
  // Do stuff here
});
Philippe Plantier
  • 7,964
  • 3
  • 27
  • 40
0

We ended up using rate limiter which provided the rate limiting we needed right out of the box. https://www.npmjs.com/package/limiter

Bill
  • 4,614
  • 13
  • 77
  • 132