4
def get_secret():
    secret_name = "--secret-name-here--"
    region_name = "--region-here--"

    # Create a Secrets Manager client
    session = boto3.session.Session()
    print("B")
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name
    )
    print(client)
    # In this sample we only handle the specific exceptions for the 'GetSecretValue' API.
    # See https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_GetSecretValue.html
    # We rethrow the exception by default.

    try:
        print("D")
        get_secret_value_response = client.get_secret_value(
            SecretId=secret_name
        )
        print("E")

I'm trying to access a secret in SecretsManager from a lambda that's within a VPC. The lambda has been configured with a NAT gateway so it is able to reach the public internet. I've tested this with a requests.get call.

client = session.client(
        service_name='secretsmanager',
        region_name=region_name
    )

This part runs just fine and I get back a SecretsManager client. However...

get_secret_value_response = client.get_secret_value(
            SecretId=secret_name
        )

This section times out. The lambda has these permissions: GetSecretValue, DescribeSecret, ListSecretVersionIds on the relevant resource, so I'm not sure what's going on. Any help is appreciated!

josephcrawley
  • 41
  • 4

1 Answers1

2

A Lambda function in a VPC does not have Internet access, because it is never assigned a public IP. The AWS SecretsManager API is on the public Internet, not in your VPC, so by default your Lambda function in a VPC can't access AWS SecretsManager.

You have two options:

Mark B
  • 183,023
  • 24
  • 297
  • 295
  • I did that already, as mentioned in my question. I've tested it, and my lambda does connect out to the internet. – josephcrawley Aug 11 '22 at 16:48
  • It sounds like you haven't actually done that correctly. You state "The lambda has been configured with a NAT gateway so it is able to reach the public internet." but what does that mean exactly? Show what you have done. For example if you just put a NAT Gateway in the same subnet as the Lambda function, that's not going to work. – Mark B Aug 11 '22 at 16:50
  • You say "it is able to access the public Internet" but how have you proven that? Do you have it accessing something else on the Internet besides SecretsManager? – Mark B Aug 11 '22 at 16:51
  • In my code, before this function runs, I have a requests.get call to an external website that completes successfully. – josephcrawley Aug 11 '22 at 16:52
  • How long is the function timeout? Have you tried increasing the timeout? – Mark B Aug 11 '22 at 17:15
  • I just tested it with a timeout increased to 5 minutes, getting same result. – josephcrawley Aug 11 '22 at 17:23
  • As stated above- you need a vpce configured to actually route your request to secrets manager. Yes you can get to the internet through the NAT gateway. But you still need an internal route to the AWS network for secrets manager. Modify your security group and sub nets to get you access to ASM. – JDBennett Aug 15 '22 at 00:50
  • @JDBennett no you don't need a VPC endpoint for secrets manager if you have a route to a NAT Gateway. The Secrets Manager API is on the public Internet. If you have a NAT Gateway giving you Internet access then that is all you need. – Mark B Aug 15 '22 at 12:23
  • So here is a simple test. Actually set-up a VPCE and see if that fixes it. Not saying you need a VPCE to get to secrets manager with public internet access - but it does demonstrate the route tables are not configured correctly. – JDBennett Aug 15 '22 at 12:50
  • Adding a VPCE doesn't seem to make a difference. – josephcrawley Aug 15 '22 at 17:20
  • Honestly it sounds like there's something really broken in your VPC network configuration. Have you edited/added non-default VPC Network ACL rules? – Mark B Aug 15 '22 at 17:25
  • It's fixed! Somehow, removing an old VPCE for ASM seemed to do the trick. Don't ask why or how though. ¯\_(ツ)_/¯ Thanks for all the help – josephcrawley Aug 15 '22 at 17:54
  • "removing an old VPCE for ASM" you never mentioned you had an endpoint for ASM already in any of the details of your question. It's probably the security group on that VPCE that was the problem, but you never mentioned it even existed. – Mark B Aug 15 '22 at 17:57
  • I had no clue someone had set one up until checking due to JDBennett's advice. – josephcrawley Aug 15 '22 at 18:28