11

Say I have a express service which sends email:

app.post('/send', function(req, res) {
  sendEmailAsync(req.body).catch(console.error)
  res.send('ok')
})

this works.

I'd like to know what's the advantage of introducing a job queue here? like Kue.

wong2
  • 34,358
  • 48
  • 134
  • 179

2 Answers2

15

Does Node.js need a job queue?

Not generically.

A job queue is to solve a specific problem, usually with more to do than a single node.js process can handle at once so you "queue" up things to do and may even dole them out to other processes to handle.

You may even have priorities for different types of jobs or want to control the rate at which jobs are executed (suppose you have a rate limit cap you have to remain below on some external server or just don't want to overwhelm some other server). One can also use nodejs clustering to increase the amount of tasks that your node server can handle. So, a queue is about controlling the execution of some CPU or resource intensive task when you have more of it to do than your server can easily execute at once. A queue gives you control over the flow of execution.

I don't see any reason for the code you show to use a job queue unless you were doing a lot of these all at once.

The specific https://github.com/OptimalBits/bull library or Kue library you mention lists these features on its NPM page:

  • Delayed jobs
  • Distribution of parallel work load
  • Job event and progress pubsub
  • Job TTL
  • Optional retries with backoff
  • Graceful workers shutdown
  • Full-text search capabilities
  • RESTful JSON API
  • Rich integrated UI
  • Infinite scrolling
  • UI progress indication
  • Job specific logging

So, I think it goes without saying that you'd add a queue if you needed some specific queuing features and you'd use the Kue library if it had the best set of features for your particular problem.

In case it matters, your code is sending res.send("ok") before it finishes with the async tasks and before you know if it succeeded or not. Sometimes there are reasons for doing that, but sometimes you want to communicate back whether the operation was successful or not (which you are not doing).

OWADVL
  • 10,704
  • 7
  • 55
  • 67
jfriend00
  • 683,504
  • 96
  • 985
  • 979
  • @wong2 - Did this answer your question? – jfriend00 Jun 23 '17 at 23:32
  • 1
    Comment from 2019: Kue is unmaintained now and it recommends to use https://www.npmjs.com/package/bull instead – Alex G.P. Aug 12 '19 at 08:31
  • What about asynchronous communication between microservices like mentioned [here](https://stackoverflow.com/a/41086984/1057791) and [here](https://stackoverflow.com/a/50466235/1057791)? it seems that a queue will be more appropriate than making a direct HTTP request to the node server, don't you think? – BornToCode May 20 '20 at 16:43
  • 1
    @BornToCode - A queue has specific situations where it makes sense because you want or need specific features the queue has. It is not something that you generically need or want for all communication with microservices. – jfriend00 May 20 '20 at 17:02
  • What if the operation has a fire-and-forget nature, e.g. the send email service is called from another microservice which doesn't need to wait (or 'await') to know the result, it's just want to continue with its internal flow after dispatching the request to send email? – BornToCode May 20 '20 at 17:17
  • 1
    @BornToCode - There's no need for a queue to do a fire and forget call. – jfriend00 May 20 '20 at 21:15
  • 1
    @BornToCode - Please write your own question if you have something specific you want to ask about. Comments on a different question are not made to discuss your specific issue. – jfriend00 May 20 '20 at 21:59
7

Basically, the point of a queue would simply be to give you more control over their execution.

This could be for things like throttling how many you send, giving priority to other actions first, evening out the flow (i.e., if 10000 get sent at the same time, you don't try to send all 10000 at the same time and kill your server).

What exactly you use your queue for, and whether it would be of any benefit, depends on your actual situation and use cases. At the end of the day, it's just about controlling the flow.

samanime
  • 25,408
  • 15
  • 90
  • 139