1

I have a use case where I want to call a downstream API for all records in S3.

The API has a rate limit of 250 TPS only, so I can call the API only 250 times in one second.

Now, the records can arrive in S3 all at the same time (within few seconds). The records will be partitioned in multiple smaller files (say 1000 records each). I was thinking to keep a lambda which is triggered by S3 PUT which puts these records in a SQS queue.

There will be another lambda which polls the queue and calls the downstream API.

S3 -> lambda-1 -> AQS -> lambda-2(calls downstream API)

How can I make sure that the lambda-2 reads only 250 messages from the queue per second?

Is there any alternate design which I can look into if this won't work.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Let us see.
  • 81
  • 8
  • 1
    You can limit the lambda concurrency and the lambda batch size - and even optionally sleep / wait within the lambda code. – luk2302 Aug 18 '22 at 09:25
  • How long would the Lambda function run for? You can control concurrency so it stays within the limit. – John Rotenstein Aug 19 '22 at 01:15
  • The lambda would run for 2-3 seconds only. Are there any downsides of using reserved concurrency if there are millions of records to process from a queue? – Let us see. Aug 19 '22 at 06:07

0 Answers0