An analog of async.queue for Async/Await functions

Question

I am modernizing some code. It has a piece to load database implemented as:

var customerQueue = async.queue(insertCustomer, DATABASE_PARALLELISM);
customerQueue.drain = function() {
    logger.info('all customers loaded');
    airportCodeMappingQueue.push(airportCodeMappings);
}

The function insertCustomer used to written with callbacks. I changed it to async/await, as a part of code modernization.

Now, think that I wrote an equivalent of async.queue as:

let customerQueueElements = [];
var customerQueue = {};
customerQueue.push = (customers) => {
  customers.forEach(customer => {
    customerQueueElements.push(insertCustomer(customer))
  });
}

const processQueue = async (queue, parallelism) => {
  for (let i = 0; i < queue.length; i += parallelism) {
    for (let j = 0; j < parallelism; j++) {
      let q = []
      if (queue[i + j]) {
        q.push(queue[i + j])
      }
      await Promise.all(q)
    }
  }
}

I am able now to do await ProcessQueue(customerQueue, DATABASE_PARALLELISM), but the syntax is bad, and I am keeping a visible named variable for each queue.

What would be a good way to handling this?

Also, drain() should be hooked-up to then, right ?

The way you have written this will essentially run your `processQueue` *synchronously* even though it is doing it with `async` promises. The `Promise.all(q)` will perform correctly but every iteration of the `for ()` will be halted until `Promise.all(q)` returns. Is this what you are intending? — Deryck, Apr 29 '18 at 14:37
"*the syntax is bad, and I am keeping a visible named variable for each queue*" - you should be able to fix that by wrapping the whole thing in a (module factory) function that returns (exports) only the thing that you need to use the queue. — Bergi, Apr 29 '18 at 14:45
the context was just to grab his attention in case he thought the loop would proceed without delay as one might when taking old code and converting to new `async/await`. — Deryck, Apr 29 '18 at 14:47
JavaScript has only one thread. Hence, nothing runs truly in parallel. I believe that this thing, if I understand how `async.queue` works, will create the `parallelism` number of async requests, much like the original. Once, those are done, it will run this thing again. Are you familiar with `async` package? Am I wrong here? — Moshe Shmukler, Apr 29 '18 at 14:49

score 0 · Answer 1 · answered Apr 29 '18 at 20:31

@Bergi is correct, as far as direction. I put together a work-in-process version:

module.exports = function () {
  module.internalQueue = []
  module.func = undefined
  module.parallelism =  1

  const process = async () => {
    for (let i = 0; i < module.internalQueue.length; i += module.parallelism) {
      for (let j = 0; j < module.parallelism; j++) {
        let q = []
        if (module.internalQueue[i + j]) {
          q.push(module.func(module.internalQueue[i + j]))
        }
        await Promise.all(q)
      }
    }
    module.internalQueue = []
    module.drain()
  }

  module.queue = (func, parallelism = 1) => {
    module.func = func
    module.parallelism = parallelism
    return module
  }

  module.push = async (customers) => {
    module.internalQueue.push(customers)
    await process()
  }

  module.drain = () => {}

  return module
}

It is not exactly correct, yet. The signatures are similar to the async package, but my version shares the queue among all instances.

Need to figure out an easy way of creating an instance for each function with a separate e.g. "local" queue. Then, it will basically work like the original one.

Will update as I have progress.

An analog of async.queue for Async/Await functions

1 Answers1