Rate limiting request-promise in for loop

Question

I'm trying to to limit the rate at which I'm firing GET requests to an outside API using promises but I'm having difficulty getting it working. In my scenario, I'm using 'request-promise' module and I need to send GET requests for 175 items from an API (one request for each item ID). The API has a rate limit of 40 requests per 10 seconds so my throttle needs to be 250ms per request. I am trying to send a request inside loop for each item ID, like:

files.forEach(function (file, i) {
    console.log("The item ID is " + file.match(re)[1]);
    client.send(new APIClient.requests.getItem(file.match(re)[1]))
    .then((item) => {
      ...
    })
    .catch((error) => {
      console.error(error);
      // Use fallback
    });
    ...

Here is a snippet of my API client return a request promise (rp) with a 250ms timeout and no callback:

const rp = require('request-promise');
const rp_errors = require('request-promise/errors');
...
send(request, callback) {
...
    return rp(options)
               .then(this._parseResponse)
               .then((response)=> {
                  return new Promise( (resolve) => setTimeout(function(){
                    if (callback) { return callback(null, response); }
                    return resolve(response);
                  }, 250));
                })
                .catch(rp_errors.StatusCodeError,((error) => {
                    throw new errors.ResponseError(request, error.statusCode, error.message);
                  }
                ))
                .catch(rp_errors.RequestError,((error) => {
                    if(error.cause.code === 'ETIMEDOUT' || error.cause.code === 'ESOCKETTIMEDOUT')
                      throw new errors.TimeoutError(request, error);
                    throw error;
                  }
                ))
                .catch((error) => {
                  if (callback) {return callback(error)};
                  throw error;
                });
}

The Async doesnt work and it returns this stack trace of "429 request limit exceeded"

{ ResponseError: 429 - {"status_code":25,"status_message":"Your request count (175) is over the allowed limit of 40."}
[0]     at rp.then.then.catch (/mnt/c/Users/ridhwaan/Source/homehost/lib/api-client.js:52:19)
[0]     at tryCatcher (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/util.js:16:23)
[0]     at /mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/catch_filter.js:17:41
[0]     at tryCatcher (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/util.js:16:23)
[0]     at Promise._settlePromiseFromHandler (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/promise.js:512:31)
[0]     at Promise._settlePromise (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/promise.js:569:18)
[0]     at Promise._settlePromise0 (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/promise.js:614:10)
[0]     at Promise._settlePromises (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/promise.js:689:18)
[0]     at Async._drainQueue (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/async.js:133:16)
[0]     at Async._drainQueues (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/async.js:143:10)
[0]     at Immediate.Async.drainQueues [as _onImmediate] (/mnt/c/Users/ridhwaan/Source/homehost/node_modules/bluebird/js/release/async.js:17:14)
[0]     at runCallback (timers.js:756:18)
[0]     at tryOnImmediate (timers.js:717:5)
[0]     at processImmediate [as _immediateCallback] (timers.js:697:5)
[0]   name: 'ResponseError',
[0]   request:
[0]    Movie {
[0]      method: 'GET',
[0]      path: '/movie/24428',
[0]      timeout: 10000,
[0]      ensureHttps: false,
[0]      external_id: '24428' },
[0]   statusCode: 429 }

"*`return new Promise( (resolve) => setTimeout(function(){ if (callback) { return callback(null, response); } return resolve(response); }, 250));`*" - WTF? As you seem to be using Bluebird, `delay` and `asCallback` are your friends. And don't keep a promise unsettled! — Bergi, Feb 25 '18 at 21:57

Kroney · Accepted Answer · 2018-02-26T17:57:51.057

So the main issue is that array.forEach is a synchronous function and will not wait for client.send to finish. A solution would be to use bluebird.mapSeries (http://bluebirdjs.com/docs/api/promise.mapseries.html) to map through an array and wait for each iteration to finish. Also don't forget to return the send-promise, so the mapSeries-function will know when it has resolved, so it knows when to start the next iteration:

bluebird.mapSeries(files, function(file){
  return send(...)
}

Last a suggestion would be to replace the whole .then(... setTimeout...) part with just .delay(250). Request-Promise already uses bluebird promises so you can make use of their convenience functions http://bluebirdjs.com/docs/api/delay.html. Delay will automatically resolve the value from the previous promise

return rp(options)
   .then(this._parseResponse)
   .delay(250)
   .catch(...)

Rate limiting request-promise in for loop

1 Answers1