0

I have 1000 http API requests to be made. I have kept them all as promises in an array. I want to execute them in "BATCHES" of 100 at a time - not more than that to avoid hitting any API rate-limit / throttling etc.

While bluebirdJS provides the .map() function with the concurrency option what it does is it limits the number of calls made AT A TIME. Meaning it will ensure that no more than 100 concurrent requests are being worked on at a time - as soon as the 1st request resolves, it will begin processing the 101st request - it doesn't wait for all the 100 to resolve first before starting with the next 100.

The "BATCHING" behavior i am looking for is to first process the 100 requests, and ONLY AFTER all of the 100 requests have completed it should begin with the next 100 requests.

Does BlueBirdJS provide any API out of the box to handle batches this way?

Rakib
  • 12,376
  • 16
  • 77
  • 113
  • 2
    Wouldn't breaking the request in a list of 100 items each and applying a `Promise.all` inside a `Promise.each` do ? – Prasanna Aug 31 '17 at 07:33

1 Answers1

2

You can split big urls array to an array of batches. For each batch run Promise#map which will be resolved when all async operations are finished. And run these batches in sequence using Array#reduce.

let readBatch(urls) {
  return Promise.map(url => request(url));
}

let read(urlBatches) {
  return urlBatches.reduce((p, urls) => {
    return p.then(() => readBatch(urls));
  }, Promise.resolve());
}

const BATCH_SIZE = 100;
let urlBatches = [];
for (let i = 0; i < urls.length; i+= BATCH_SIZE) {
  let batch = array.slice(i, i + BATCH_SIZE);
  urlBatches.push(batch);
}

read(urlBatches)
   .then(() => { ... }) // will be called when all 1000 urls are processed
alexmac
  • 19,087
  • 7
  • 58
  • 69
  • nice.... i had used `.mapSeries()` instead of `.reduce` to run through the batches in series.... the code is somewhat cleaner with `.mapSeries()` – Rakib Sep 06 '17 at 03:55
  • In your question you asked about solution for running a new batch, **ONLY AFTER all of the 100 requests have completed** `Bluebird#mapSeries` works differently. It runs operations in series, not in parallel [docs](http://bluebirdjs.com/docs/api/promise.mapseries.html). – alexmac Sep 06 '17 at 15:58
  • But that's what we are trying to do, right? To run one batch after another batch in series/sequence - so `Bluebird#mapSeries()` does exactly that as you cited the docs. The promises INSIDE a batch can, however, happen in parallel – Rakib Sep 07 '17 at 06:23
  • No, it's different. `Array#reduce` runs batches in series, but async operations in _a batch_ in parallel. `Bluebird#mapSeries()`... I don't understand how it can be used to run batches, it can run only async operations in a batch, and only in series. – alexmac Sep 07 '17 at 09:13