21

I have a lambda function written in Python that uses a couple of heavyweight dependencies (NumPy, pandas, goodtables, etc.) and is also connected to a VPC (for access to a Postgres RDS instance)

The cold start execution time of this function is huge (16.2 seconds) when it's executed after a while (> 4-6 hours)

However, if I update the function code and invoke it a second time (shortly after the first execution), the cold start execution time reduces drastically (3 seconds)

If I invoke the function again without updating it so it's a warm start, the execution time goes down even further (313 ms)

I suspect the first cold start (16.2 secs) is when Lambda sets up an ENI for access to VPC resources and the ENI is reused during the second cold start (3 secs) so the time taken to re-create the ENI is avoided.

I am trying to optimize the cold start time of this function and want it to start from scratch to see how fast it can execute when starting completely cold (i.e. no ENI + cold start).

Is there a way to do this and do it repeatedly?

Vinayak
  • 1,103
  • 3
  • 18
  • 40
  • 2
    What if you create a *new function* instead of just a new (or "updated") version? The ENI management operations and associated logic are almost entirely opaque, but you can see them in retrospect in your CloudTrail logs. – Michael - sqlbot Mar 17 '19 at 16:23
  • @Michael-sqlbot I was hoping to avoid creating a new function every time I wanted a cold start. However, from the output of `aws ec2 describe-network-interfaces | grep -C5 ` I can see whether or not an ENI is attached and the time of attachment (if one is attached). If I could detach/delete this ENI, do you think it'll force Lambda to re-create the ENI on next invocation? – Vinayak Mar 17 '19 at 16:58
  • You can try it but I doubt it will work as you expect. Lambda doesn't expect you to pull interfaces out from under it and you may find this just stops the function from working altogether. Not sure about your objection to creating a new function. The whole thing can be automated using aws-cli. – Michael - sqlbot Mar 17 '19 at 18:05
  • @Michael-sqlbot I didn't know I could automate it using aws-cli. Is it possible to create a new function using code from an existing function without having to upload a new ZIP? – Vinayak Mar 18 '19 at 10:31
  • 1
    [`aws lambda create-function`](https://docs.aws.amazon.com/cli/latest/reference/lambda/create-function.html) accepts `--code` to fetch the zip file from S3 or `--zip-file` to upload the zip directly. – Michael - sqlbot Mar 19 '19 at 00:17
  • @Michael-sqlbot awesome! Thanks! – Vinayak Mar 19 '19 at 07:44
  • @Vinayak did you ever find a solution to this? – MrfksIV Apr 15 '20 at 09:43

3 Answers3

36

You can toggle the memory up, save and reset it back again.

You can also add a new environment variable.

This forces all existing warm lambda's to be disposed and a new cold start on the next invocation of the lambda.

Damian Haynes
  • 384
  • 3
  • 3
  • 1
    If I remember correctly, this didn't work (see my question) as the ENI wasn't recreated – Vinayak Jun 29 '19 at 18:50
  • Thanks - I did see the question, in practice we use this method all the time, maybe the current ENI takes some time to dispose? – Damian Haynes Jul 08 '19 at 05:12
  • Changing the memory to a lower value and then back up worked for me. – TylerW Jun 10 '20 at 15:21
  • 2
    Thanks to this answer, over the past year I've been using the method of either adding or just *modifying* one of the environment variables. I typically just create a new one such a forceRestart and increment the counter. – ebol2000 Jul 17 '20 at 18:07
  • I don't now if it's specific to golang runtime, but modifying environment doesn't cause cold start anymore. – smirnoff Jan 13 '23 at 17:02
1

Instead of just modifying the code, you can try to publish new version of you lambda function for testing purposes. According to AWS, each time you publish a new version of your lambda function, all the containers in which your function is running are destroyed and then recreated, which should force complete cold start.

Matus Dubrava
  • 13,637
  • 2
  • 38
  • 54
  • If you meant running [`update-function-code`](https://docs.aws.amazon.com/cli/latest/reference/lambda/update-function-code.html), that's what I currently do and it'll cold start my lambda but the cold start time is less than 5 seconds so I suspect the ENI is reused and not re-created. – Vinayak Mar 17 '19 at 14:08
  • No, I'm not. Let me try doing that and I'll get back to you. – Vinayak Mar 17 '19 at 14:22
  • Nope, that didn't do anything apart from publish a new version of the function. Cold start time was not affected (2.5 seconds). – Vinayak Mar 17 '19 at 15:54
  • @MatusDubrava updating the function code also publishes a new version -- just not a numbered version. `$LATEST` points to the newly-published, otherwise-anonymous version. – Michael - sqlbot Mar 17 '19 at 16:20
0

I was wondering the same thing and while you could "throttle" the reserved count to zero in a testing scenario, that most likely won't be a viable option in a production one. For that, take a look at the answers in either Force Discard AWS Lambda Container or Restarting AWS lambda function to clear cache.

ebol2000
  • 1,195
  • 11
  • 16