1

If I have an event listener to any network connection where the underlying messages from the protocol are guaranteed to be ordered. It could be NodeJS' net TCP socket connection or a rabbitmq AMPQ connection with a high prefetch value.

I want to find a way to serially process them in the order that the messages arrive from the event listener.

Here's the rough sketch of the solution I've come up with exploiting NodeJS' single threaded event loop behaviour. I believe it will always work from a theoretical standpoint unless I've missed something.

Are they any drawbacks to this approach and can we do better? What I can see immediately is that the stack size of the recursive call can grow and exceed the limit if we receive a lot of messages in quick succession.

beingProcessed = false;
queue = new Queue(); // synchronous in memory queue 

.on("messageFromTCPSocketOrAMPQ", (msg) => { // sync callback
  // very important that the callback here does no async work prior to inserting into the 
  // queue to guarantee that the next message won't be pushed first
  queue.push(msg); // in memory queue
  processSerially(msg);
})

// sync function
processSerially() {
 if(beingProcessed || queue.size() === 0) {
   return;
 }
 beingProcessed = true;

 // this function can be made never throw depending on the use case
 doSomeTaskAsyncWithMessage(queue.pop(), () => {
   beingProcessed = false;
   // this recursive call in the callback is the reason why this works
   processSerially(); 
 })
}
Louis Christopher
  • 308
  • 3
  • 17
  • 2
    If `doSomeTaskAsyncWithMessage` is actually asynchronous ([like setTimeout](https://stackoverflow.com/q/24631041/1048572)), it [will not overflow the stack](https://stackoverflow.com/q/29144361/1048572). – Bergi Apr 09 '21 at 20:02
  • Yes, it's definition is `async doSomeTaskAsyncWithMessage(msg) {} ` let's say calling a database etc – Louis Christopher Apr 09 '21 at 20:12
  • If it's an `async` function and returns a promise, you shouldn't pass a callback though :-) – Bergi Apr 09 '21 at 20:38
  • Ah, so I'd make `processSerially` async and then `await doSomeTask(msg)` followed by a pseudo recursive `unawaited processSerially`? Thanks, maybe you can answer the question and I'll accept it? – Louis Christopher Apr 10 '21 at 05:56
  • 1
    I wouldn't necessarily make `processSerially` an `async` function, but at least write `doSomeTask(queue.pop()).catch(…).then(() => { beingProcessed = false; processSerially(); })`. However, if you *do* make it `async function processSerially`, you wouldn't even need (pseudo) recursion but could directly write a `while` loop. – Bergi Apr 11 '21 at 17:07

1 Answers1

-1

You will NOT (edit: thanks bergi, see his comment) run into a Maximum call stack size exceeded error if you keep doing the recursive method. Here is a way with a while loop where the queue pushes are decoupled from the processing.

queue.on("messageFromTCPSocketOrAMPQ", (msg) => { // sync callback
  queue.push(msg); // in memory queue
})

while (true) {
  if (queue.size() > 0) {
    doSomeTaskAsyncWithMessage(queue.pop())
  }
}
richytong
  • 2,387
  • 1
  • 10
  • 21
  • Thanks, I don't believe the code example is complete in your answer, you will need to break out of the loop in the else condition or the thread will stay busy and you will need to await inside the loop or they will run out of order. – Louis Christopher Apr 09 '21 at 19:24
  • 2
    No, it will not run into a stack overflow. – Bergi Apr 09 '21 at 20:03