2

I noticed that when i updated the secret, it takes sometime before the lambda is able to retrieve the updated secret value. I wonder if there is some caching happening during lambda invocation.

alegria
  • 1,290
  • 14
  • 23
  • Have you an example of your function? – Chris Williams Jun 08 '20 at 14:41
  • 1
    You can take Lambda out of the equation to understand if Secrets Manager itself caches secrets or you are otherwise subject to eventual consistency concerns. Use the awscli to retrieve a secret, modify the secret, and retrieve it again. – jarmod Jun 08 '20 at 17:11
  • Thanks @jarmod, that's a good suggestion. I can confirm that the change seems to reflect in the aws cli output instantly. – alegria Jun 08 '20 at 18:06
  • @mokugo-devops it's a pretty complicated codebase, and I'm not very familiar with lambda yet, but i will try to isolate that specific code to test easily. – alegria Jun 08 '20 at 18:09
  • 2
    Basically if you intialise the secrets manager client outside of the function and try to retrieve the secret outside of the function, it will not be executed every time – Chris Williams Jun 08 '20 at 18:10
  • 3
    FYI that @mokugo-devops is referring to Lambda warm starts vs. cold starts. – jarmod Jun 08 '20 at 18:12
  • thanks @jarmod i think that answers it! It's doing a warm start and using the execution context which contains the previous value. Eventually, after some time, it gets a cold start and is able to get the new secret value. – alegria Jun 08 '20 at 20:26

1 Answers1

3

The only builtin caching I'm aware of in lambda function is the execution context reuse, which is documented here.

Take advantage of execution context reuse to improve the performance of your function. Initialize SDK clients and database connections outside of the function handler, and cache static assets locally in the /tmp directory. Subsequent invocations processed by the same instance of your function can reuse these resources. This saves execution time and cost.

To answer your question, if you fetch the secrets outside the function handler, then it will take some time to fully update in the execution context.

jellycsc
  • 10,904
  • 2
  • 15
  • 32
  • I think this answers it. The lambda is a flask app. The aws secret is in an object and is initialised only once during the execution of the app. When the aws secret was updated, it might be that the call executions are still using the current execution context and the previously retrieved secret value. – alegria Jun 08 '20 at 20:20
  • Eventually, after being idle for some time, it gets a cold start and is able to get the new secret value. – alegria Jun 08 '20 at 20:28