NodeJS Express API - Ticketing/Queue System

Question

Rephrased at the end

NodeJS communicates with other APIs through GRPC.

Each external API has its own dedicated GRPC connection with Node and every dedicated GRPC connection has an upper bound of concurrent clients that it can serve simultaneously (e.g. External API 1 has an upper bound of 30 users).

Every request to the Express API, may need to communicate with External API 1, External API 2, or External API 3 (from now on, EAP1, EAP2 etc) and the Express API also has an upper bound of concurrent clients (e.g. 100 clients) that can feed the EAPs with.

So, how I am thinking of solving the issue:

1. A Client makes a new request to the Express API.

2. A middleware, queueManager, creates a Ticket for the client (think of it as a Ticket that approves access to the System - it has basic data of the Client (e.g. name))

3. The Client gets the Ticket, creates an Event Listener that listens to an event with their Ticket ID as the event name (when the System is ready to accept a Ticket, it yields the Ticket's ID as an event) and enters a "Lobby" where, the Client, just waits till their ticket ID is accepted/announced (event).

My issue is that I can't really think of how to implement the way that the system will keep track of the tickets and how to have a queue based on the concurrent clients of the system.

Before the client is granted access to the System, the System itself should:

1. Check if the Express API has reached its upper-bound of concurrent clients -> If that's true, it should just wait till a new Ticket position is available
2. If a new position is available, it should check the Ticket and find out which API it needs to contact. If, for example, it needs to contact EAP1, it should check how many current clients use the GRPC connection. This is already implemented (Every External API is under a Class that has all the information that is needed). If the EAP1 has reached its upper-bound, then NodeJS should try again later (But, how much later? Should I emit a system event after the System has completed another request to EAP1?)

I'm aware of Bull, but I am not really sure if it fits my requirements. What I really need to do is to have the Clients in a queue, and:

1. Check if Express API has reached its upper-bound of concurrent users

• If a position is free, pop() a Ticket from the Ticket's array

2. Check if the EAPx has reached its upper-bound limit of concurrent users

• If true, try another ticket (if available) that needs to communicate with a different EAP
• If false, grant access

Edit: One more idea could be to have two Bull Queues. One for the Express API (where the option "concurrency" could be set as the upper bound of the Express API) and one for the EAPs. Each EAP Queue will have a distinct worker (in order to set the upper bound limits).

REPHRASED

In order to be more descriptive about the issue, I'll try to rephrase the needs.

A simple view of the System could be:

I have used Clem's suggestion (RabbitMQ), but again, I can't achieve concurrency with limits (upper-bounds).

So,

1. Client asks for a Ticket from the TicketHandler. In order for the TicketHandler to construct a new Ticket, the client, along with other information, provides a callback:

TicketHandler.getTicket(variousInfo, function () {
    next();
  })

The callback will be used by the system to allow a Client to connect with an EAP.

2. TickerHandler gets the ticket:

   i) Adds it to the queue

   ii) When the ticket can be accessed (upper-bound is not reached), it asks the appropriate EAP Handler if the client can make use of the GRPC connection. If yes, then asks the EAP Handler to lock a position and then it calls the ticket's available callback (from Step 1) If no, TicketHandler checks the next available Ticket that needs to contact a different EAP. This should go on until the EAP Handler that first informed TicketHandler that "No position is available", sends a message to TicketHandler in order to inform it that "Now there are X available positions" (or "1 available position"). Then TicketHandler, should check the ticket that couldn't access EAPx before and ask again EAPx if it can access the GRPC connection.

Answer 1

From your description I understand what follows:

• You have a Node.js front-tier. Each Node.js box needs to be limited to up to 100 clients
• You have an undefined back-tier that has GRPC connections with the boxes in the front-tier (let's call them EAPs). Each EAP <-> Node.js GRPS link is limited to N concurrent connections.

What I see here are only server-level and connection-level limits thus I see no reason to have any distributed system (like Bull) to manage the queue (if the Node.js box dies there is no one able to recover the HTTP request context to offer a response to that specific request - therefore when a Node.js box dies responses to its requests are not more useful).

This being considered I would simply create a local queue (as simple as an array) to manage your queuing.

Disclaimer: this has to be considered pseudo-code what follows is simplified and untested

This may be a Queue implementation:

interface SimpleQueueObject<Req, Res> {
  req: Req;
  then: (Res) => void;
  catch: (any) => void;
}


class SimpleQueue<Req = any, Res = any> {

  constructor(
    protected size: number = 100,
    /** async function to be executed when a request is de-queued */
    protected execute: (req: Req) => Promise<Res>,
    /** an optional function that may ba used to indicate a request is
     not yet ready to be de-queued. In such case nex request will be attempted */
    protected ready?: (req: Req) => boolean,
  ) { }

  _queue: SimpleQueueObject<Req, Res>[] = [];
  _running: number = 0;

  private _dispatch() {
    // Queues all available
    while (this._running < this.size && this._queue.length > 0) {
      // Accept
      let obj;
      if (this.ready) {
        const ix = this._queue.findIndex(o => this.ready(o.req));
        // todo : this may cause queue to stall (for now we throw error)
        if (ix === -1) return;
        obj = this._queue.splice(ix, 1)[0];
      } else {
        obj = this._queue.pop();
      }
      // Execute
      this.execute(obj.req)
        // Resolves the main request
        .then(obj.then)
        .catch(obj.catch)
        // Attempts to queue something else after an outcome from EAP
        .finally(() => {
          this._running --;
          this._dispatch();
        });
      obj.running = true;
      this._running ++;
    }
  }

  /** Queue a request, fail if queue is busy */
  queue(req: Req): Promise<Res> {
    if (this._running >= this.size) {
      throw "Queue is busy";
    }

    // Queue up
    return new Promise<Res>((resolve, reject) => {
      this._queue.push({ req, then: resolve, catch: reject });
      this._dispatch();
    });
  }

  /** Queue a request (even if busy), but wait a maximum time
   * for the request to be de-queued */
  queueTimeout(req: Req, maxWait: number): Promise<Res> {
    return new Promise<Res>((resolve, reject) => {
      const obj: SimpleQueueObject<Req, Res> = { req, then: resolve, catch: reject };
      // Expire if not started after maxWait
      const _t = setTimeout(() => {
        const ix = this._queue.indexOf(obj);
        if (ix !== -1) {
          this._queue.splice(ix, 1);
          reject("Request expired");
        }
      }, maxWait);
      // todo : clear timeout
      // Queue up
      this._queue.push(obj);
      this._dispatch();
    })
  }

  isBusy(): boolean {
    return this._running >= this.size;
  }

}

And then your Node.js business logic may do something like:

const EAP1: SimpleQueue = /* ... */;
const EAP2: SimpleQueue = /* ... */;

const INGRESS: SimpleQueue = new SimpleQueue<any, any>(
  100,
  // Forward request to EAP
  async req => {
    if (req.forEap1) {
      // Example 1: this will fail if EAP1 is busy
      return EAP1.queue(req);
    } else if (req.forEap2) {
      // Example 2: this will fail if EAP2 is busy and the request can not
      // be queued within 200ms
      return EAP2.queueTimeout(req, 200);
    }
  }
)

app.get('/', function (req, res) {
  // Forward request to ingress queue
  INGRESS.queue(req)
    .then(r => res.status(200).send(r))
    .catch(e => res.status(400).send(e));
})

Or this solution will allow you (as requested) to also accept requests for busy EAPs (up to a max of 100 in total) and dispatch them when they become ready:

const INGRESS: SimpleQueue = new SimpleQueue<any, any>(
  100,
  // Forward request to EAP
  async req => {
    if (req.forEap1) {
      return EAP1.queue(req);
    } else if (req.forEap2) {
      return EAP2.queue(req);
    }
  },
  // Delay queue for busy consumers
  req => {
    if (req.forEap1) {
      return !EAP1.isBusy();
    } else if (req.forEap2) {
      return !EAP2.isBusy();
    } else {
      return true;
    }
  }
)

Please note that:

• in this example, Node.js will start throwing when more than 100 concurrent requests are received (it is not unusual to throw a 503 while throttling)
• Be careful when you have more throttling limits (Node.js and GRPC in your case) as the first may cause the seconds starvation (think about receiving 100 requests for EAP1 and then 10 for EAP2, Node.js will be full with EAP1 requests and will refuse EAP2 ones all do EAP2 is doing nothing)