Implementing a queue of function calls that runs in parallel to the rest of the code

Question

I'm currently building a program in JavaScript that is making requests of the google sheets API based on activity occurring in a Discord Server (messaging app). However, I've been running into the API RateLimits in cases where multiple users do the same action at the same time, causing too many API Requests in too short of a time.

My idea to get around this is to implement a parallel queue of async function calls, so that whenever I want to make a request of the google API, I queue that function call, and another function or thread or something will keep checking this queue and if there is a function available, it will run that function, wait a little bit, and then check the queue again, and so on.

I'm struggling to figure out how to do this in regular asynchronous (async/await) programming. I've been referring to the following posts/pages, but they all seem focused on a predefined queue that is then dequeued in order - I want to be able to keep adding to the queue even after the functions have started being run. How do I store javascript functions in a queue for them to be executed eventually Semaphore-like queue in javascript? https://www.codementor.io/@edafeadjekeemunotor/building-a-concurrent-promise-queue-with-javascript-1ano2eof0v

Any help or guidance would be very appreciated, thank you!

Answer 1

The simplest option would be to have a queue of promise-returning functions and poll it periodically.

Example:

let queue = []

async function poll() {
    console.log('POLL, queue.length=', queue.length)
    if (queue.length) {
        let result = await queue.shift()()
        console.log('RESULT', result.id)
    }
    setTimeout(poll, 1000)
}

let n = 0

function test() {
    let getter = () => fetch(
        'https://jsonplaceholder.typicode.com/todos/' + (++n) 
    ).then(r => r.json())

    queue.push(getter)
    
}


poll()

<button onclick="test()">click many times</button>

Answer 2

Besides what T.J. Crowder already did mention about true parallelism in JavaScript there is this special requirement of wanting to continuously being able to add to the queue (enqueue) after the (de)queue based processing had started. Therefore I doubt there will be any solution based entirely on promises.

Thus, in case one does not want to go for permanently running "background" tasks based on setInterval/setTimeout, one has to implement an approach capable of handling callbacks.

One way was to e.g. implement a request class which is capable of dispatching its own (custom) events. It should be possible for both node and Web Api environments (browsers) since the latter provides/supports EventTarget and the former features packages for it.

Possible implementation details are as follows.

Any request-queue can be instantiated with the batchSize-parameter's integer-value where this parameter indicates the desired amount of fetch requests which are going to participate in a single all-settled promise handling.

Once such a promise is settled - regardless of any fetch promise's status - at least one of both custom queue events will be thrown, either the 'new-batch' or the 'rejected' event type or both event types. Each custom-event's detail payload will feature its type specific data, e.g. a resolved array for the former and a rejected array for the latter case.

Regarding the handling of rejected api calls (the list of rejected fetch URLs),

• one could implement the handling callback in a way that it
  • collects/accumulates such data until a certain threshold where one then would pass this data to the request-queue's fetch method again.
• one too could implement functionality which prevents fetching the same url(s) again and again up to a maximum retry count.

But the latter proposed features should not be part of the queue implementation.

// helper functions for logging and creating a list
// of api requests.
function logRequestQueueEvent({ type, detail }) {
  console.log({ [ type ]: { detail } });
}
function createFetchArrayFromBoundApiCallCount() {
  let { callCount = 0, fetchSize = 12 } = this;

  const result = Array
    .from({ length: fetchSize })
    .map((_, idx) =>
      `https://jsonplaceholder.typicode.com/photos/${ idx + callCount + 1 }`
    );
  this.callCount = callCount + fetchSize;

  return result;
}

// initializing of the main example which uses an
// instance of a custom implemented request-queue
// class which is capable of both
//  - fetching continuously
//  - and dispatching events.
function main() {
  const requestQueue = new ContinuouslyFetchingRequestQueue(5);

  // a queue instance's three available event types one can subsribe to.
  requestQueue.addEventListener('new-fetch', logRequestQueueEvent);
  requestQueue.addEventListener('new-batch', logRequestQueueEvent);
  requestQueue.addEventListener('rejected', logRequestQueueEvent);

  // as for handling rejected api calls (the list of rejected URLs),
  //  - one could implement the handling callback in a way that it
  //     - collects/accumulates such data until a certain threshold
  //       where one then would pass this data to the request-queue's
  //       `fetch` method again.
  //  - one too could implement functionality which prevents fetching
  //    the same url(s) again and again up to a maximum retry count.
  // but such features should not be part of the queue implementation.

  const createFetchArray = createFetchArrayFromBoundApiCallCount
    .bind({ callCount: 0, fetchSize: 12 });

  document
    .querySelector('[data-request]')
    .addEventListener('click', () =>
      // a queue instance's sole public accessible method.
      requestQueue.fetch(createFetchArray())
    );
}
main();

body { zoom: .9; margin: 0; }
button { display: block; width: 5em; margin: 10px 0; }
.as-console-wrapper { min-height: 100%!important; width: 89%; top: 0; left: auto!important; }

<script>
// helper function for creating chunks from an array.
function chunkArray(arr = [], chunkLength = arr.length) {
  chunkLength = Math.abs(chunkLength);

  const result = [];
  while (arr.length >= 1) {

    result.push(
      arr.splice(0, chunkLength)
    );
  }
  return result;
}

// `queue` instance related requests and responses handler.
function handleRequestsAndResponses(queue, fetching, addresses) {

  // for each `addresses` array create an all-settled promise ...
  Promise
    .allSettled(
      addresses.map(url => fetch(url)
        .then(response => response.json())
        .catch(error => ({ error, url }))
      )
    )
    .then(results => {
      // ... where each settled promise item either features
      // the JSON-parsed `value` or a failing `reason`.

      const resolved = results
        .filter(({ status }) => status === 'fulfilled')
        .map(({ value }) => value);

      const rejected = results
        .filter(({ status }) => status === 'rejected')
        .map(({ reason }) => reason.url);

      // since a `queue` instance features inherited
      // `EventTarget` behavior, one can dispatch the
      // above filtered and mapped response arrays as
      // `detail`-payload to a custom-event like 'new-batch'.
      queue
        .dispatchEvent(
          new CustomEvent('new-batch', {
            detail: { resolved, fetching: [...fetching] },
          }),
        );

      // one also could think about dispatching the
      // list of rejected addresses per bundled fetch
      // separately, in case there are any.

      // guard.
      if (rejected.length >= 1) {
        queue
          .dispatchEvent(
            new CustomEvent('rejected', {
              detail: { rejected },
            }),
          );
      }
    })
}

// `queue` instance related fetch/request functionality.
function createBundledFetch(queue, fetching, batchSize) {
  queue
    .dispatchEvent(
      new CustomEvent('new-fetch', {
        detail: { fetching: [...fetching] },
      }),
    );

  // decouple the `queue` related `fetching`
  // reference from the to be started request
  // process by creating a shallow copy.
  const allAddresses = [...fetching];

  // reset/mutate the `queue` related `fetching`
  // reference to an empty array.
  fetching.length = 0;

  // create an array of chunked `addresses` arrays ...
  chunkArray(allAddresses, batchSize)
    .forEach(addresses => setTimeout(

      // ... and invoke each bundled request and
      // response-batch handling as non blocking.
      handleRequestsAndResponses, 0, queue, fetching, addresses,      
    ));
}

// queue instance method implemented
// as `this` context aware function.
function addAddressListToBoundQueueData(...addressList) {

  // assure a flat arguments array (to a certain degree).
  addressList = addressList.flat();

  // guard.
  if (addressList.length >= 1) {
    const { queue, fetching, batchSize } = this;

    fetching.push(...addressList);

    // invoke the bundled fetch creation as non blocking.
    setTimeout(
      createBundledFetch, 0, queue, fetching, batchSize,
    );
  }
}

// custom request-queue class which is capable of both
//  - fetching continuously
//  - and dispatching events.
class ContinuouslyFetchingRequestQueue extends EventTarget {
  constructor(batchSize) {
    super();

    batchSize = Math
      .max(1, Math.min(20, parseInt(batchSize, 10)));

    const fetching = [];
    const queue = this;

    // single/sole public accessible instance method.
    queue.fetch = addAddressListToBoundQueueData
      .bind({ queue, fetching, batchSize });
  }
}
</script>

<button data-request>add 12 requests</button>
<button onclick="console.clear();">clear console</button>