0

I have a AWS Lambda with limited concurrency set that is consuming messages from a AWS SQS of type FIFO, and I just learned that the "messages in flight"-property of the queue does not reflect the number of concurrent lambdas processing the messages.

However, do I need to worry that the Default Visibility Timeout of the queue will time out before a lambda or in the middle of a lambda just having received a message? Or rather is the Default Visibility Timeout counted from when the message was turned into "in-flight", or is it counted from when the lambda start processing the message?

Judging from the Amazon SQS visibility timeout documentation it almost sound like the former, but I feel that it still a bit vague and it would be helpful if someone knew for sure.

Adelost
  • 2,343
  • 2
  • 23
  • 28
  • 1
    What you’re saying here is correct. A message could theoretically timeout and be consumed by another process while it’s in the middle of being consumed. This is because SQS guarantees at least once delivery. And any system using it should not be affected by messages being consumed more than once. https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html#standard-queues-at-least-once-delivery – bryan60 Apr 09 '20 at 12:03
  • @bryan60 that is unfortunate then, but thanks for clearing that out. Can you think of any workaround at the moment? Right now I am thinking of from scratch trying to implement lambdas polling from the queue, but it feels like this should be a common enough use-case for it to already have been solved in some other way. – Adelost Apr 09 '20 at 13:40
  • jobs being run more than once is usually a risk in queue systems by their nature. As clearly a lot could go wrong that prevents the systems from properly communicating. It’s basically a guarantee with enough messages to process. Generally, you want to make your system not care if it runs the same job twice. It suggests a flaw in your design if this is not the case. If you’re really not able to have messages be processed more than once, you’re probably using the wrong tool. Your two basic options are you risk having things run more than once or you risk having things not get run at all. – bryan60 Apr 09 '20 at 14:12

0 Answers0