multiple nodejs consumers for sqs queue?

Question

I have a SQS queue with thousands of messages. I cannot consume them on lambda due to size of function (using puppeteer) so I've moved my consumer to EC2 instance(s). As this is POC, I did a very basic implementation of consumer using this (https://github.com/bbc/sqs-consumer) library and launched the same process using pm2 multiple times.

import { SQSClient } from '@aws-sdk/client-sqs';
import { Consumer } from 'sqs-consumer';

const sqs = new SQSClient({
  apiVersion: '2012-11-05',
  region: 'us-east-1',
  credentials: {
    accessKeyId: #@@@#
    secretAccessKey: 232131321312
  },
});

const app = Consumer.create({
  queueUrl: EnvsEnum.QUEUE_URL,
  terminateVisibilityTimeout: true,
  waitTimeSeconds: 20,
  handleMessage: async (message) => {
... my handler here doing some work on the message 
  },
  shouldDeleteMessages: true,
  region: 'us-east-1',
  sqs,
});

app.on('error', (err, item) => {
...
});

app.on('processing_error', (err) => {
...
});
console.log('starting app...');
app.start();

I expected to see multiple messages in flight, but on SQS dashboard I only see 1 constantly. What could be the reason? Maybe some good guide to implementing a proper consumer if I went about it completely wrong?

Update

I solved the problem by realizing that I was sending the same MessageGroupId to the queue - fixing that makes the consumers work in parallel as expected.

multiple nodejs consumers for sqs queue?

0 Answers0