1

I know SQS has the maxReceiveCount and once the message has been consumed and failed this amount of times, then the message is moved to the DLQ. That's working currently.

However, there are certain errors which I would want to go back in my source queue indefinitely e.g. if we get a 5XX error from an API we are invoking. This is very likely to be temporary, and so we would want to retry these until they are successful.

Now if these 5XX failed messages do go to the DLQ, it's not the end of the world, but it requires a manual redrive.

Is there a known solution or aws feature which allows for conditionally moving messages to the DLQ depending on the error received?

I'm not sure this is possible. I've read the documentation and can't find anything that hints at how to achieve this.

Ben Parker
  • 13
  • 4
  • 1
    No such feature, unless you develop custom solution for that. – Marcin Jan 04 '23 at 09:22
  • 1
    I'm assuming you are using a lambda that would make the call against that API you mentioned. If that's the case, based on the response code, you could decide to put back the message in the queue (as if it is a new message). I'd also recommend to have a custom counter as part of your message to not retry infinite number of time and be stuck in a situation where your lambda process the same message over and over. Also please consider that you can set Maximum receives to 1000. You could throw an error based on the response code and let SQS moving the message to the DLQ once it reached 1000. – brushtakopo Jan 04 '23 at 09:31
  • @AnthonyB. This is something you could post as a “real” answer to this question. :) – lxg Jan 04 '23 at 10:37
  • @AnthonyB. Yeah I'm using a lambda. The idea of putting the message back as a new one is really interesting. But wouldn't this lead to potentially duplicated messages, since the original failing message will be added back to the queue to reprocess until it hits the maxReceiveCount limit? I suppose I could process the original message as if it were processed correctly so the message doesn't go back on the queue, but that makes me feel a bit dirty.. – Ben Parker Jan 04 '23 at 10:40

1 Answers1

1

Since you didn't provide much detail about the way you process the message, I'm assuming you are using a lambda that would make the call against that API you mentioned.

If that's the case, based on the response code, you could decide to put back the message in the queue (as if it is a new message) and return success so that the message is not duplicated.

I'd also recommend to have a custom counter as part of your message to not retry infinite number of time and be stuck in a situation where your lambda process the same message over and over.

A preferable approach would be to set Maximum receives to 1000 for DLQ. You could throw an error based on the response code in your lambda and let SQS moving the message to the DLQ once it reached 1000. 1,000 it is quite a lot of retries..

brushtakopo
  • 1,238
  • 1
  • 3
  • 16