1

I'm using S3 Batch Operations to invoke a lambda to load a bunch of data into elasticsearch. Reserved concurrency is set to 1 on the lambda while experimenting with the right amount of concurrency.

Strangely when testing I see S3 calls the lambda a few times back-to-back, and then makes no more invocations for about 5 minutes. For example:

Lambda invocations

Any ideas how to avoid this delay? There are no other batch operations happening on the account and priority is set to "10". Maybe it has something to do with the reserved concurrency value? Although according to the documentation:

When the job runs, Amazon S3 starts multiple function instances to process the Amazon S3 objects in parallel, up to the concurrency limit of the function. Amazon S3 limits the initial ramp-up of instances to avoid excess cost for smaller jobs.

Thank you

user2923125
  • 77
  • 1
  • 7
  • Was there a particular reason you set reserved concurrency to 1? Is there a reason not to run concurrent Lambdas here? – jarmod Oct 08 '21 at 14:36
  • Note that priority 10 doesn't make a job run sooner. It just tries (with no guarantees) to run it before other, lower priority jobs (of which you have none). – jarmod Oct 08 '21 at 14:42
  • I see no guarantees about the delays. I have experience only with S3 event notifications, which are usually processed in a few seconds, but sometimes it may take a few minutes. – gusto2 Oct 08 '21 at 21:30

1 Answers1

0

Remove the reserved concurrency limit of 1, or set it to a much higher value.

Setting 1 as the value for reserved concurrency will restrict the maximum number of concurrency invocations for your Lambda function to just one.

It also limits your function from using concurrency from the unreserved pool which is normally 1000 concurrent invocations.

The result of this is throttling of your function to just 1 instance running at any given time.

Now, the reason why some go through sometimes after one another is probably that the function is able to process some items quickly enough before S3 Batch Operations sends the next batch of data, staying in line with only 1 instance running.

Synchronous requests arriving in excess of the reserved capacity limit will fail with a throttling error, which is exactly what's happening here.

Ermiya Eskandary
  • 15,323
  • 3
  • 31
  • 44