12

I'm writing a flask API in pycharm. When I run my code locally, requests using boto3 to get secrets from secrets manager take less than a second. However, when I put my code on an EC2, it takes about 3 minutes (tried in both t2.micro and m5.large).

At first I thought it could be a Python issue, so I ran it in my EC2s through the awscli using:

aws secretsmanager get-secret-value --secret-id secretname

It sill took about 3 minutes. Why does this happen? Shouldn't this in theory be faster in an EC2 than in my local machine?

EDIT: This only happens when the EC2 is inside a VPC different than the default VPC.

rodrigocf
  • 1,951
  • 13
  • 39
  • 62
  • When you say "inside a VPC different than the default VPC", does the 'slow' VPC have any unusual configuration, such as going via a corporate network instead of directly out to the Internet via an Internet Gateway? Merely being a 'default' VPC should not make a difference. – John Rotenstein May 07 '18 at 00:23
  • Did you ever happen to figure out what the problem here was? We're having the same problem when running the request locally, but it is fine on AWS. – LTAcosta Oct 09 '18 at 18:45

4 Answers4

23

After fighting with this same issue on our local machines for almost two months, we finally had some forward progress today.

It turns out the problem is related to IPv6.

If you're using IPv6, then the secrets manager domain will resolve to an IPv6 address. For some reason the cli is unable to make a secure connection using IPv6. After it times out, the cli falls back to IPv4 and then it succeeds.

To verify if you're resolving to an IPv6 address, just ping secretsmanager.us-east-1.amazonaws.com. Don't worry about the ping response, you just want to see the IP address the domain resolves to.

To fix this problem, you now have 3 options:

  1. Figure out your networking issues. This could be something on your machine or router. If in an AWS VPC, check your routing tables and security groups. Make sure you allow outbound IPv6 traffic (::/0).
  2. Reduce the cli connect timeout to make the IPv6 call fail faster. This will make the IPv4 fallback happen sooner. You may want give a better timeout value, but the general idea is to add something like this: --cli-connect-timeout 1
  3. Disable IPv6. You can either disable IPv6 on your machine/router altogether, or you can adjust your machine to prefer IPv4 for this specific address (See: https://superuser.com/questions/436574/ipv4-vs-ipv6-priority-in-windows-7).

Ultimately, option 1 is the real solution, but since it is so broad, the others might be easier.

Hopefully this helps someone else maintain a bit of sanity when they hit this.

LTAcosta
  • 500
  • 3
  • 9
  • 1
    There is one other option if you are running inside an EC2 VPC. You can add a VPC endpoint to Secrets Manager and select local DNS. This will create an endpoint in your VPC with an IP4 only address and modify local DNS in the VPC to make the Secrets Manager endpoint resolve to that address. Best of all traffic routes directly to Secrets Manager. – JoeB May 01 '20 at 00:05
5

I had this issue when working from home through the Cisco AnyConnect VPN client. Apparently it blocks anything IPv6.

The solution for me was to disable IPv6 altogether on my laptop.

To do so for macos:

networksetup -setv6off Wi-Fi  # wireless
networksetup -setv6off Ethernet  # wired

To re-enable:

networksetup -setv6automatic Wi-Fi  # wireless
networksetup -setv6automatic Ethernet  # wired
AArias
  • 2,558
  • 3
  • 26
  • 36
  • 1
    This solved my local issue with Fortinet VPN instantly. Interestingly, toggling IPv6 off and on again also helps. – dirkjot Aug 27 '20 at 15:44
0

I ran the following commands from my own computer and from an Amazon EC2 t2.nano instance in the ap-southeast-2 region:

aws secretsmanager create-secret --name foo --secret-string 'bar' --region ap-southeast-2
aws secretsmanager get-secret-value --secret-id foo --region ap-southeast-2
aws secretsmanager delete-secret --secret-id foo --region ap-southeast-2

In both cases, each command returned within a second.

Additional:

To test your situation, I did the following (in the Sydney region):

  • Created a new VPC using the VPC Wizard (just a public subnet)
  • Launched a new Amazon EC2 instance in the new VPC, with a Role granting permission to access Secrets Manager
  • Upgraded AWS CLI on the instance (the installed version didn't know about secretsmanager
  • Ran the above commands

They all returned immediately.

Therefore, the problem lies with something to do with your instances or your VPC.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • I created a 3 brand new EC2s outside of my vpc, could the fact that the first EC2s I tried it with were inside a VPC? – rodrigocf May 07 '18 at 00:17
  • What do you mean by "outside of my VPC"? Where they are located should not matter, as long as they can reach the Internet. If the command eventually worked, then that doesn't sound like a problem. You could turn on the **debug** option (`aws --debug secretsmanager...`) to see whether there is a communication problem. – John Rotenstein May 07 '18 at 00:21
  • In the default VPC it has the regular behavior of less than a second. The non-default VPC only has a route table between the 0.0.0.0/0 and the internet gateway, so no weird configurations. – rodrigocf May 07 '18 at 00:35
  • Just ran it and I can see it hang in ```2018-05-07 00:46:37,700 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): secretsmanager.us-east-1.amazonaws.com ```, then literally exactly 3 minutes after moves to ```2018-05-07 00:49:37,855 - MainThread - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "POST / HTTP/1.1" 200 447``` – rodrigocf May 07 '18 at 00:52
  • Any thoughts? The problem seems to be inside of the VPC only and for 3 exact minutes. Everyt single call to secrets manager outside of the VPC works fine. – rodrigocf May 08 '18 at 23:04
  • I ran some tests, see additional info above. – John Rotenstein May 09 '18 at 00:40
  • The 3 minute time out is the signature of the IPV6 problem described above. Secrets Manger presents a dual stack endpoint (that is to say has both IPV4 and IPV6 addresses). In addition, the older AWS CLI used an old version of urllib (recently moving to urllib3). When your instance has a globally routable IPV6 address, the older urllib would try the endpoint addresses in order; IPV6 first. With a one minute time out and 3 IPV6 addresses this gives the 3 minute delay. Either add an IPV6 egress gateway to your VPC, or remove the IPV6 address block (assuming that is the problem). – JoeB Jan 04 '19 at 23:31
  • If you can't adjust the VPC connectivity, then a possible workaround is to set the connection timeout (default of 60 seconds) something shorter. I'm using '--cli-connect-timeout 5', and now I'm down to "only" 30 seconds to read a single secret. – Ken Pronovici Mar 18 '23 at 00:56
0

I made the hotspot from my phone and it worked

Alex Pătroi
  • 39
  • 6