I am working on a NodeJS & RabbitMQ where -
- A Master NodeJS Service (one instance) Pushes the data in a queue.
- Slave NodeJS Services (multiple instances) consume the data and process it.
- Default exchange is being used.
The slave service is running in PM2 Cluster Mode, which means I have got 8 instances for the slave service running.
My expectation was, as the master service starts pushing in data through the queue, The slave services should start consuming them asynchronously.
As an example, if master service is pushing 10 jobs through the queue and if each job is taking 5 seconds to complete, it takes slaves 50 seconds to finish the job.
This is completely defeating the purpose of using multiple slaves as I would nomaly want slaves to pick up 8 jobs at a time.
As per the RabbitMQ dashboard, above setup creates -
- 9 connections ( 1 master + 8 slaves )
- 9 channels (1 master + 8 slaves )
- 1 queue
The whole setup uses default exchange.
My question is -
Why are the slaves not able to read data from the queue asynchronously?
Even is I set noAck to true the next item in the queue will not get picked up until curent item is processed
My intention is to scale up the queue consumption rate by user multiple slave instances but I think I am missing something here.
Here's the codebase -
const rabbitMq = require("amqplib");
class RabbitMQClient {
async connect() {
this.connection = await rabbitMq.connect("amqp://admin:password@localhost");
this.channel = await this.connection.createChannel();
await this.channel.assertQueue("TEST_QUEUE");
}
}
// MASTER CODE (This Runs In Fork Mode - 1 Instance)
const master_client = new RabbitMQClient();
master_client.connect().then(() => {
// sending in 10 messages
for (let index = 1; index <= 10; index++) {
const data = Buffer.from(index.toString(), "utf8");
master_client.channel.sendToQueue("TEST_QUEUE", data);
}
});
// SLAVE CODE (This Runs In Cluster Mode - 8 Instances)
const slave_client = new RabbitMQClient();
// connect to rabbitmq
slave_client.connect().then(() => {
// consume the messages
slave_client.channel.consume("TEST_QUEUE", (data) => {
// timeout to add delay
setTimeout(() => {
RabbitMQClient._channel.ack(data);
}, 5000);
});
});
The slave output -
33|slave | 2020-11-02 13:19:09.293 +00:00: recieved message - 1 (13-19-09)
34|slave | 2020-11-02 13:19:14.293 +00:00: recieved message - 2 (13-19-14)
35|slave | 2020-11-02 13:19:19.299 +00:00: recieved message - 3 (13-19-19)
36|slave | 2020-11-02 13:19:24.299 +00:00: recieved message - 4 (13-19-24)
37|slave | 2020-11-02 13:19:29.300 +00:00: recieved message - 5 (13-19-29)
38|slave | 2020-11-02 13:19:34.299 +00:00: recieved message - 6 (13-19-34)
39|slave | 2020-11-02 13:19:39.301 +00:00: recieved message - 7 (13-19-39)
40|slave | 2020-11-02 13:19:44.301 +00:00: recieved message - 8 (13-19-44)
33|slave | 2020-11-02 13:19:49.299 +00:00: recieved message - 9 (13-19-49)
34|slave | 2020-11-02 13:19:54.300 +00:00: recieved message - 10 (13-19-54)
If you notice, different slaves are picking up the messages in round-robin fashion but they are working synchronously.
Thanks!
