1

I'm using an AWS Lambda function that is triggered from an SNS event trigger to consume from an SQS queue. When the Lambda function executes, it pulls 10 messages from the queue, processes them, pulls another 10, and so on and so forth - up to a certain time limit that's coded into the Lambda function (less than the max of 5 minutes, obviously).

It's my understanding that a Lambda function triggered by an SNS event is one-to-one, is that correct? In other words, one SNS event won't trigger multiple Lambda functions (up to the maximum concurrent execution limit). There's no scaling based on load.

Are there any other potential solutions, leveraging Lambda, that would let me consume from SQS as frequently/fast as possible? I had considered trying to auto-scale my Lambda functions by leveraging CloudWatch alarms (and SNS event triggers) based on SQS queue size, but it seems like those alarms can fire, at most, every 5 minutes. I've also considered developing a master Lambda function that can automatically execute (many) slave Lambdas based on querying the queue size.

I understand that the more optimal design may be to leverage Kinesis instead of SNS. I may consider incorporating Kinesis in the future, but let's just pretend that Kinesis is not an option at this time.

littleK
  • 19,521
  • 30
  • 128
  • 188
  • 1
    I'm not aware of any guarantees here except that SNS will retry, per configured policy, and then fail. But, even if it did invoke Lambda twice (which would be extremely rare, I suspect), any SQS messages being processed by Lambda #1 would be invisible to Lambda #2 (per visibility timeout). – jarmod Jan 28 '18 at 17:42
  • Not sure what you mean by *"In other words, one SNS event won't trigger multiple Lambda functions (up to the maximum concurrent execution limit). There's no scaling based on load."* The Lambda function will be invoked for each message published to the SNS topic... even if the function invocation from the previous publish is still running, up to the concurrency limit, which sounds like the opposite of what you are asserting, but I'm not sure. – Michael - sqlbot Jan 28 '18 at 21:29
  • @Michael-sqlbot, I was trying to assert that the number of Lambda executions is equal to the number of SNS trigger events - but multiple instances of the Lambda functions can be invoked and executing at one time (up to the max concurrent execution limit). – littleK Jan 28 '18 at 21:59
  • Right. SNS will invoke the function once per event message published to the topic. – Michael - sqlbot Jan 28 '18 at 23:19

2 Answers2

0

There is no best way to do this. One approach (which you've kind of already mentioned) is to use CloudWatch and schedule a Lambda function to run every minute (that's the minimum schedule time for Lambda). This Lambda function will then look for new SQS messages and invoke other Lambda functions to handle new message(s). Here is a very good article for that use case: https://cloudonaut.io/integrate-sqs-and-lambda-serverless-architecture-for-asynchronous-workloads/

Personally, I do not recommend triggering your Lambda by SNS for this use case, because SNS doesn't give a full guarantee for delivery and recommend sending the SNS notifications to SQS - which does not solve your problem. From the FAQ's:

[...] If it is critical that all published messages be successfully processed, developers should have notifications delivered to an SQS queue (in addition to notifications over other transports).

Source: https://aws.amazon.com/sns/faqs/

s.hesse
  • 1,900
  • 10
  • 13
0

For this kind of processing, instead of SQS if you push messages to Kinesis Stream you should be able to flexibly process(In batches of needed size) the messages.

Note: If you use SQS, after triggering a Lambda function through SNS (or using a Scheduled Lambda), it can invoke inner Lambda functions to check the queue where multiple concurrent inner Lambdas are spawned. However the problem is that its not practical to process SQS items in batches.

Ashan
  • 18,898
  • 4
  • 47
  • 67