1

I have an SQS queue linked to a deadletter queue via a redrive policy. Terraform sample:

  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.deadletter_queue.arn
    maxReceiveCount     = 10
  })

The queue processing is implemented with exponential backoff. If processing is successful the message is simply deleted. If processing fails it retries with an initial VisibilityTimeout of 30 seconds, which doubles each time, the final retry timeout (between 9th and 10th attempt) is roughly 2 hours.

My question is, after the final retry, when does the message get sent to the deadletter queue? Immediately after the 10th attempt or does it have to wait out the VisibilityTimeout (of about 4 hours)?.

My concern is that it seems redundant to put a visibility timeout on a message if it's just going to be sent straight to the deadletter - this delays alerts for example.

wilmol
  • 1,429
  • 16
  • 22

2 Answers2

1

If you have 4 hour visibility timeout, after 10 th attempt SQS will have to wait 4 hours before it can deem the attempt unsuccessfully. SQS can't assume that the attempt failed after 1 minute, if it may take 4 hours to process the message. SQS does not know what are you doing with the messages and has no means of checking if your application failed to process the message before visibility timeout expires.

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • Thanks, makes sense. Also think we're using `VisibilityTimeout` incorrectly - its more about giving time to process a message, not implementing a delay for retry? Now looking at delay queue to replace this. – wilmol Oct 04 '21 at 01:37
0

Lets say you are running 2 consumers, A and B.

Visibility timeout is a time period which start right after "consumer A" receives the message. For this duration, "consumer B" is not able to receive the message (message is reserved for the "consumer A"). So visibility timeout's purpose is to "hide" message from other consumers to avoid duplication, while one of the consumers already working on it (also see aws doc here).

About the question, after the final retry, message is immediately moved to dead letter queue. It doesn't wait for a visibility timeout to be over, because it won't have the chance to be consumed again. Message is moved to dead letter queue and removed from source queue.

mozh
  • 33
  • 4