5

I've got an SQS queue set up with multiple Lambdas configured as triggers. While this is allowed by AWS's API it's not clear what the intended behavior is.

For example, is one message pulled off the queue and sent to each Lambda? Or instead does each Lambda act as a polling service which pulls messages off the queue separately?

From my testing it appears to be the latter, but I can't for the life of me find this documented anywhere. The latter also doesn't make a ton of sense to me given AWS Lambda's concurrency functionality.

Matthew 'mandatory' Bryant
  • 197
  • 2
  • 7
  • 1
    If this is supported, it shouldn't be. The idea behind SQS is to be 1-to-1 or 1 job to 1 worker. As @mlu points out below the typical design pattern for this use case is to fan-out using SNS. – Levi Oct 15 '18 at 23:50

1 Answers1

7

SQS message can only be read once from the queue and while it's being processed by one consumer it will be invisible to other consumers. If you've got multiple lambdas each will get their share of the messages but one message won't be delivered to more than one lambda at a time.

SQS Messages may be re-inserted into the queue if the processing Lambda fails (if the Lambda is triggered by the queue) or if the message is not deleted within the visibility timeout (if the Lambda is polling the queue).

If you need a single message processed by multiple Lambdas concurrently you can either:

  1. fan-out the SQS message to SNS that will have multiple subscribers (e.g. multiple Lambdas, all processing the same message), or
  2. use Kinesis stream instead of SQS. Kinesis supports multiple consumers of the same message.

Hope that helps :)

MLu
  • 24,849
  • 5
  • 59
  • 86