11

Has anyone else solved the following problem?
I have SNS topic filled with events from S3 and there is Lambda function which is subscribed on this topic and when thousand of events are put to this topic, lambda function is throttled because of exceeding the limit of concurrency.
I don't want to request a limit increase for concurrent executions but I would decrease concurrent consuming from the topic, but I didn't find information how to do it. Thanks.

nikolayandr
  • 171
  • 1
  • 1
  • 11

1 Answers1

16

A couple of options regarding SNS:

1) SNS Maximum Receive Rate

Set the SNS Maximum Receive Rate. This will throttle the SNS messages sent to a subscriber, but may not be a great option if you have so many messages that they will be discarded before they can be processed. From the documentation:

You can set the maximum number of messages per second that Amazon SNS sends to a subscribed endpoint by setting the Maximum receive rate setting. Amazon SNS holds messages that are awaiting delivery for up to an hour. Messages held for more than an hour are discarded.

If you're only getting thousands of events at a time, setting the Maximum Receive Rate to Lambda's default concurrent execution limit of '100' might be worth a try.

As @kndrk notes, this throttling is currently only available for HTTP/HTTPS subscribers to the SNS topic. To work around this, you can expose your lambda function via AWS API Gateway and subscribe that endpoint to the SNS topic, rather than the lambda function directly.

2) Process from SQS

Subscribe an SQS queue to the SNS topic and process messages from the queue, rather than directly from the sns topic. A single invokation of SQS ReceiveMessage can only handle 10 messages at a time, so that may be easier for you to throttle.

It is also worth noting that you can publish S3 Events directly to AWS Lambda.

Anthony Neace
  • 25,013
  • 7
  • 114
  • 129
  • Thanks for quick answer. I tried the solution when lambda was triggering by S3 but the same problem has occurred, therefore I decided to try combination S3->sns->lambda. I'm going to try your first option, it may be suitable for me. **P.S.** I want to process events from S3 immediately because in most cases the load is not so high, but if I understood correctly, lambda cannot be triggered by events from SQS. Only way it is to schedule executing of lambda function for consuming from SQS – nikolayandr Nov 06 '16 at 20:34
  • That's correct -- for SQS you would essentially be polling SQS RecieveMessage from lambda on a schedule instead of plugging it into a specific trigger. It sounds like from your comment that SNS Maximum Receive Rate would be a nice fit for your use case. Hope it works out for you! – Anthony Neace Nov 06 '16 at 20:38
  • 2
    Unfortunately this only applies to HTTP(s) endpoints. Lambda subscribers aren't throttled at all. – knrdk Dec 08 '16 at 19:28
  • Thanks @knrdk! Wasn't following that initially but I re-read and that's a great point! Think we can get around that with an API Gateway endpoint, I've updated answer with a workaround. – Anthony Neace Dec 08 '16 at 19:44
  • Yup, it works for API Gateway endpoints, so that's what we ended up doing. – knrdk Dec 08 '16 at 21:12
  • Please note throttle is available for SNS [up to 3 times](https://docs.aws.amazon.com/lambda/latest/dg/invoking-lambda-function.html#supported-event-source-sns) – Nicolas Labrot Jan 15 '18 at 12:40
  • Regarding `SNS Maximum Receive Rate`, the quote doesn't exist anymore in the document. Does it mean that the message will stay in the SNS until it's delivered? – Mooncrater Jun 09 '21 at 08:18
  • @AnthonyNeace Have asked a question [here](https://stackoverflow.com/questions/67900337/what-happens-to-sns-self-throttled-messages-which-are-not-delivered-for-a-long). – Mooncrater Jun 09 '21 at 08:26