26

I am trying to trigger lambda execution to execute an item on a FIFO queue. Other than polling, what options do we have to accomplish that? We just learned that we cannot directly trigger a lambda execution from a FIFO queue, which is only supported from the standard queue at this time. I also learned that we cannot subscribe an SNS topic to a FIFO queue – which is only supported on the standard queue as well.

Has anybody found a work around for this yet until Amazon releases an update?

845614720
  • 756
  • 1
  • 7
  • 22
  • 5
    **November 2019:** AWS Lambda now supports Lambda triggers. See: [AWS Lambda Supports Amazon SQS FIFO (First-In-First-Out) as an Event Source](https://aws.amazon.com/about-aws/whats-new/2019/11/aws-lambda-supports-amazon-sqs-fifo-event-source/) – John Rotenstein Nov 20 '19 at 01:17

3 Answers3

24

Your only option here is to poll the queue at certain intervals.

You can create a CloudWatch event trigger that invokes a lambda function (lets say every 10 minutes) which polls the queue and does the processing.

Update

This is now possible. SQS FIFO queues can now be specified as an event source to your Lambda Function.

https://aws.amazon.com/blogs/compute/new-for-aws-lambda-sqs-fifo-as-an-event-source/

Asdfg
  • 11,362
  • 24
  • 98
  • 175
  • 3
    I'm interested to know why Amazon does it like this, with my limited knowledge I believe it has something to do with the lambdas spawning multiple instances when called? So if you had them on a FIFO SQS you couldn't be sure that they items are being processed in order because they run async? Correct me if I'm wrong. – 845614720 Nov 21 '18 at 19:06
  • 2
    Yes. That is what I think is the reason. If there are more messages in your SQS, more lambda functions will be invoked behind the scene means more lambda instances polling the same queue so it becomes difficult to mange the sequential execution. – Asdfg Nov 21 '18 at 19:12
  • and that's weird, because what's the complexity of having the lambda function returning something like status 200 which would invoke the next message processing ? In my opinion if we have the standard SQS queue then we can't be sure how many lambdas are gonna be invoked for the incoming messages – Tony Mar 21 '19 at 10:43
  • Ugh. [Message groups](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/using-messagegroupid-property.html) could completely solve this problem -- and for the use case I'm exploring, _would_ solve this problem. I need duplicate elimination and FIFO _within_ a message group, but interleaving message groups in different lambdas is just fine (and desirable, actually). – Andrew May 07 '19 at 14:48
  • @Andrew - Using message groups requires the use of FIFO. How do you trigger a lambda on a FIFO Queue? If you cannot then how do you consume this queue? – Anup Kalbalia Oct 14 '19 at 21:07
  • @AnupKalbalia Unless this limit changes, Lambda [event sources](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html) only work with standard queues, not FIFO. So, you need to trigger the lambda some other way, such as the CloudWatch event trigger that the answer mentions. My comment related to a use case I have in which I don't need all FIFO features, but I can't use message groups without FIFO, so I'm stuck doing things the difficult way. – Andrew Oct 14 '19 at 21:28
  • @Andrew - are you using CloudWatch event trigger? Does that allow to process the queue as soon as there is an item in the queue? I guess you can not schedule the trigger to run with a frequency of less than a minute. I am planning to use an SNS to invoke the lambda. – Anup Kalbalia Oct 17 '19 at 12:14
3

For anyone that is interested AWS Now Supports Lambda Triggers from FIFO SQS as of yesterday. Thanks @JohnRotenstein for pointing that out.

845614720
  • 756
  • 1
  • 7
  • 22
-4

SQS event sourcing does not do anything than pulling queue every second with long polling.

If you want everything in order; - Create a Cloudwatch event trigger for every second (as Asdfg suggested) - Set concurrency limit to 1 - Receive messages with long polling to reduce the cost

Eren
  • 1