AWS secret manager time out sometimes

Question

I am fetching a secret from secret manager on a lambda. The request fails sometimes. Which is totally strange, it is working fine and couple of hours later I check and I am getting time out.

def get_credentials(self):
    """Retrieve credentials from the Secrets Manager service."""

    boto_config = BotoConfig(connect_timeout=3, retries={"max_attempts": 3})
    secrets_client = self.boto_session.client(
        service_name="secretsmanager",
        region_name=self.boto_session.region_name,
        config=boto_config,
    )
    secret_value = secrets_client.get_secret_value(SecretId=self._secret_name)

    secret = secret_value["SecretString"]

I try to debug the lambda and later seems to be working again, without any change, those state changes happen in hours. Any hint why that could happen?

Traceback (most recent call last):
  File "/opt/python/botocore/endpoint.py", line 249, in _do_get_response
    http_response = self._send(request)
  File "/opt/python/botocore/endpoint.py", line 321, in _send
    return self.http_session.send(request)
  File "/opt/python/botocore/httpsession.py", line 438, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://secretsmanager.eu-central-1.amazonaws.com/"

Is your Lambda function associated with a VPC? If so, is there any particular reason _why_ you associated it with a VPC? Also, is it associated with _multiple subnets_? It is possible that some of these subnets are Public Subnets and some are Private Subnets and it is randomly associating with a Public Subnet -- this will actually _not_ give Internet access to the function. — John Rotenstein, Mar 29 '22 at 07:15

score 2 · Answer 1 · answered Jul 19 '22 at 08:35

You are using the legacy retry mode (is the default one in boto3), which has very limited functionality as it only works for a very limited number of errors.

You should try changing your retry strategy to something like Standard retry mode or Adaptative. In that case your code would look like:

from botocore.config import Config as BotoConfig
boto_config = BotoConfig(
    connect_timeout=3,
    retries={
        "max_attempts": 3,
        "mode":"standard"
    }
)
secrets_client = self.boto_session.client(
    service_name="secretsmanager",
    region_name=self.boto_session.region_name,
    config=boto_config,
)

This is amazing, thank you so much! I have literally been trying to solve this problem for months. It's so intermittent, which made it maddening to track down. Note: the `standard` retry mode uses a default of 3 `max_attempts`, so that line is redundant. — Nick K9, Jul 23 '22 at 16:31

AWS secret manager time out sometimes

1 Answers1