1

I'm trying to implement a worker queue to handle burst of messages. Every few days/weeks I get a burst of ~5,000 messages that need to be processed in a reasonable time (each message can be proceeded in about 1-2 min). Ideally if I could run 5,000 lambdas simultaneous the whole process should take 1-2 min but my account is limit to 1,000 concurrency lambda.

I was planning to create an SQS as event source to lambda but I can't see any Throttling mechanism? I can set reserved concurrency on the lambda but it will be deducted from my account limit and it sound "expensive" to keep those lambda idle for an event that can occur only one per few weeks.

Is there a way to throttling SQS? or maybe I need to choose other service to implement such queue?

user3369398
  • 227
  • 2
  • 9
  • You can request lambda concurrency limit increase from aws support if that would solve your issue. – Marcin Mar 04 '21 at 01:08
  • @Marcin I don't want to allow it to use all my account concurrency or other lambda will start to fail such lambda triggered by API gateway etc.. I need throttling, let say allow only 100 workers at a time. – user3369398 Mar 04 '21 at 01:18

1 Answers1

0

In that case, the reserved concurrency is the way to limit number of your lambda functions that can run at the same time:

Reserved concurrency – Reserved concurrency creates a pool of requests that can only be used by its function, and also prevents its function from using unreserved concurrency.

It does not cost extra. I think you are confusing reserved concurrency with provisioned concurrency:

When enabled, Provisioned Concurrency keeps functions initialized and hyper-ready to respond in double-digit milliseconds.

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • by writing "expensive" I mean that if I choose reserved concurrency of let say 200. my account concurrency pool will drop to max of 700. So I will have 200 slots idle for a task that occur only one per few weeks? That doesn't make any sense. – user3369398 Mar 04 '21 at 01:39
  • @user3369398 There is no other way to reduce concurrency of a lambda function. If this does not suit you, you can look into other compute solutions, ecs or ec2, with autoscaling based on SQS message number. Then you can have 1 docker container only processing your messages, and when you have more messages, it will autoscale. – Marcin Mar 04 '21 at 01:45