SQS consumer is missing a bunch of messages

Question

Question I have noticed an issue where sqs consumer possibly misses the messages in the queue. Around an year ago, it was perfectly fine and this started to happen before a few months.

Here's my code Sending messages into SQS Queue:

AWS.config.update({
  region: config.aws.sqs.region,
  accessKeyId: config.aws.sqs.access_key,
  secretAccessKey: config.aws.sqs.secret_key
});

var sqs = new AWS.SQS({ apiVersion: '2012-11-05' });

module.exports.publishMessage = (txHash, eventLogs) => {
  return new Promise(async (resolve, reject) => {
    try {

      let sent_count = 0;

      logger.info(null, `Publishing events ${eventLogs.length}`)

      eventLogs.forEach((event) => {
        event.transaction_id = txHash;
        const param = {
          MessageBody: JSON.stringify(event),
          MessageGroupId: uuid.v4(),
          MessageDeduplicationId: uuid.v4(),
          QueueUrl: config.aws.sqs.node_consumer_queue_url
        }
        logger.info(null, `Sending events ${JSON.stringify(param)}`)

        sqs.sendMessage(param, function (err, data) {
          if (err) logger.error(null, err); // an error occurred
          else {
            logger.info(null, `Published: ${JSON.stringify(data)}`);

            // Debug purposes
            sent_count += 1;
            console.log(sent_count, "Sent count!");
            resolve();
          }
        });
      });
    } catch (e) {
      console.log('e', e);
      resolve();
    }
  })
}

Receiving messages from the queue

AWS.config.update({
    region: config.aws.sqs.region,
    accessKeyId: config.aws.sqs.access_key,
    secretAccessKey: config.aws.sqs.secret_key
});

let total_count = 0;

// Create our consumer
const app = Consumer.create({
    queueUrl: queueUrl,
    handleMessageBatch: async (messages) => {
        total_count += messages.length;
        return new Promise(async (resolve, reject) => {
            async.each(messages, async function (message, callback) {
                // console.log(message);
                console.log(total_count, "Received count!");

                resolve();
            });
            console.log("Execution complete!!");

        });
    },
    sqs: new AWS.SQS({
        httpOptions: {
            agent: new https.Agent({
                keepAlive: true
            })
        }
    }),
    batchSize: 10,
    waitTimeSeconds: 5
});

app.on('error', (err) => {
    logger.error(null, `Error in consumer: ${JSON.stringify(err)}`, err, err.message);
});

app.on('message_received', (msg) => {
    logger.info(null, `SQS Message Received`)
});

Sending messages in queue is working correctly whereas receiving is the part where works incorrects. When I sent 20 messages, it received 12, upon sending 10 messages, it received 8. Something weird is going on.

screenshots

Additional context Version: Latest

I have tried using handleMessage instead of handleMessageBatch but it misses even more messages.

SQS consumer is missing a bunch of messages

0 Answers0