1

I an using the npm package: sqs-consumer to poll for messages in a queue. Whenever a new message comes in I want it to create a subfolder in an S3 bucket. The issue that I am having is that even after the message is received and then deleted from the queue the function is still getting executed which is creating 100's of subfolders. I only want one subfolder per message.

Here is the S3 bucket function

const createSubDirectory = (s3BucketName, s3ObjectKey) => {
  var params = { Bucket: `${s3BucketName}`, Key: `${s3ObjectKey}`, ACL: "public-read", Body: "body does not matter" };

  try {
    s3.putObject(params);
    console.log("Error creating the folder: ", err);
  } catch (e) {
    console.log(`Subdirectory successfully created: ${s3BucketName}/${s3ObjectKey}`);
  }
};

Below is my main function:

const app = Consumer.create({
    queueUrl: process.env.SQS_QUEUE_URL,
    WaitTimeSeconds: 20,
    VisibilityTimeout: 600, // 10 min wait time for anyone else to process.
    shouldDeleteMessages: true,
    batchSize: 1,
    pollingWaitTimeMs: 10000,
    messageAttributeNames: ["All"],
    handleMessage: (message) => {
      let event = JSON.parse(message.Body);
      console.log(event);
      sqs.receiveMessage(
        {
          MaxNumberOfMessages: 1,
          MessageAttributeNames: ["All"],
          QueueUrl: process.env.SQS_QUEUE_URL,
        },
        function (err, data) {
          if (err) {
            console.log("Receive Error", err);
          } else {
            console.log("-------------- MESSAGE RECEIVED -------------");
             
            let s3ObjectKey = `${parentDirectory}/${AWS.util.uuid.v4()}/`
            createSubDirectory(s3BucketName, s3ObjectKey);

            sqs.deleteMessage({ QueueUrl: process.env.SQS_QUEUE_URL, ReceiptHandle: message.ReceiptHandle }, function (err, data) {
                if (err) {
                  console.log("Delete Error", err);
                } else {
                  console.log("Message Deleted: ", data.ResponseMetadata.RequestId);
                }
              });
          }
        }
      );
    },
    sqs: new AWS.SQS({
      apiVersion: "2012-11-05",
      region: "us-east-1",
      httpOptions: {
        agent: new https.Agent({
          keepAlive: true,
        }),
      },
    }),
  });

  app.on("error", (err) => {
    console.error(err.message);
  });

  app.on("processing_error", (err) => {
    console.error(err.message);
  });

  app.start();
kaise white
  • 67
  • 4
  • The same is happening to me, the message is deleted from the queue but it seems the same message is being processed over and over again. – Falcon Stakepool Mar 28 '23 at 05:43

1 Answers1

0

In handleMessage handler you no need to call sqs.receiveMessage and its inside methods. No need to set batchSize, as it by default process one message at a time.

Just put your create directory code in handleMessage.

I hope it will solve your problem.

BK Suraj Wasan
  • 1
  • 1