27

According to documentation the maximum SQS message retention period(MessageRetentionPeriod) is 14 days. After that time message will be deleted from the queue.

Is any way with SQS to not lose these messages after their retention period expired? For example, it is not clear or is it possible to use Dead Letter Queue for this purpose?

BinaryButterfly
  • 18,137
  • 13
  • 50
  • 91
alexanoid
  • 24,051
  • 54
  • 210
  • 410
  • I am curious why this is an issue for you. It is not within the typical use case for a message queue to leave messages in the queue for hours, much less several days. – Michael - sqlbot Aug 03 '16 at 02:39
  • 6
    I have millions of batch tasks which was estimated to take 1-2 months, I also need to keep the messages in queue after 14 days! – Jingpeng Wu Feb 23 '18 at 22:01

1 Answers1

16

Well 14 days is the max limit you can keep the message. After 14 days you can move that message to S3 Bucket for backup. Also there is a hack you can do with DLQ.

Here is a quick hack where you send that message back to the main queue. This is definitely not the best or recommended option.

  1. Set the main SQS queue as the DLQ for the actual DLQ with Maximum Receives as 1.
  2. View the content in DLQ (This will move the messages to the main queue as this is the DLQ for the actual DLQ)
  3. Remove the setting so that the main queue is no more the DLQ of the actual DLQ
BinaryButterfly
  • 18,137
  • 13
  • 50
  • 91
Piyush Patil
  • 14,512
  • 6
  • 35
  • 54
  • Thanks for your answer. One more question - in a what form messages will be backed up at S3 Bucket ? Is it an automatic process ? – alexanoid Aug 02 '16 at 17:43
  • 1
    No it is not Automatic you will have to use Cli to automate and to move that messages from SQS to S3 – Piyush Patil Aug 02 '16 at 17:45
  • But AFAIK from AWS documentation, these messages going to be deleted so how to back up them after their retention period expired ? – alexanoid Aug 02 '16 at 17:46
  • No you have to do that before the retention period – Piyush Patil Aug 02 '16 at 17:47
  • 1
    @alexanoid You can use a polled Lambda function to check DLQ and backup the data into S3 bucket. – Chamin Wickramarathna Jan 24 '19 at 06:41
  • 29
    How will this work to increase retention period? below is what is written in AWS documentation ```The expiration of a message is always based on its original enqueue timestamp. When a message is moved to a dead-letter queue, the enqueue timestamp remains unchanged. For example, if a message spends 1 day in the original queue before being moved to a dead-letter queue, and the retention period of the dl queue is set to 4 days, the message is deleted from the dl queue after 3 days. .``` source - https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html – Rishabh Dugar Aug 22 '19 at 07:49
  • Can you help me understnad which is the timestamp parameter on the message that is being referred to decided retention period. – Deva Nov 24 '21 at 10:07