3

Forgive me if this question is off-topic for this community; I couldn't find a more appropriate one and StackOverflow has always come through when I need it!

I am trying to set up a bastion host with Amazon EC2. I want the only way to connect to any of my instances to be an SSH from this bastion instance. The public subnet containing the bastion uses CIDR block 10.0.128.0/17, and the subnet containing my other instances uses CIDR block 10.0.0.0/17. I have network ACL and security group rules permitting SSH egress from the bastion to the other subnet, and permitting SSH ingress to the other subnet from the bastion. Everything should work. Unfortunately, my bastion is trying to communicate with the other instances using their public IPs, which of course is not in the 10.0.0.0/17 block, and therefore the traffic is being blocked. How can I ensure that my bastion uses private IP addresses while communicating with other instances in the private subnet? This seems like it should be the default behavior for local traffic in a VPC but apparently its not!

EDIT:

I left out some key info. The "private" instance giving me trouble is actually public; it is a Wordpress web server with public IP 52.14.20.167 (please don't spam it lol) and a custom www DNS name. However, while I want my bastion to be able to SSH into it using that DNS name, I still want all SSH traffic be local so that my security groups and network ACLs can be very restrictive. According to this AWS doc:

We resolve a public DNS hostname to the public IPv4 address of the instance outside the network of the instance, and to the private IPv4 address of the instance from within the network of the instance.

However, I think this rule only applies to the AWS-provided (IP-like) public DNS names. My custom DNS always resolves to the public IP, not the private one, as seen in the flow log from my bastion's subnet below. 10.0.128.6 is the bastion and 52.14.20.167 is the web server. Idk what the 190. and 14. addresses are. So my more educated question is, how can I have custom DNS names resolve to private IPs for a bastion host and second instance in the same VPC?

10.0.128.6  52.14.20.167    56008   22  6   7   420 1485465879  1485465996  REJECT  OK
190.173.143.165 10.0.128.6  27754   22  6   1   40  1485466241  1485466296  REJECT  OK
10.0.128.6  52.14.20.167    56012   22  6   7   420 1485466903  1485467016  REJECT  OK
190.13.10.206   10.0.128.6  28583   22  6   1   40  1485467140  1485467197  REJECT  OK
10.0.128.6  52.14.20.167    56014   22  6   7   420 1485467437  1485467557  REJECT  OK
14.158.51.244   10.0.128.6  55532   22  6   1   44  1485467500  1485467557  REJECT  OK
Rabadash8820
  • 2,328
  • 3
  • 27
  • 49
  • From the bastion host, are you using DNS names, or IP addresses, to reach those other instances? If using DNS names, to what IP addresses do those DNS names resolve on that bastion host? – Castaglia Jan 26 '17 at 22:33
  • *"This seems like it should be the default behavior for local traffic in a VPC but apparently its not!"* Unless, of course, you're diagnosing the problem incorrectly... which you are. An instance in VPC would never try to communicate with another instance's private IP using its own public IP, even if you wanted it to. Why do you think this is what is happening? – Michael - sqlbot Jan 26 '17 at 23:44
  • What is the content of your Route Tables ? – Kfactor21 Jan 26 '17 at 23:50
  • Thanks for your comments everyone, this has been very informative. @Castaglia The bastion uses DNS, and those names are resolving to the public IPs, so that must be the issue. I never would have caught that! So what is the best practice in a scenario like this, create additional A records that resolves to the private IPs? – Rabadash8820 Jan 27 '17 at 03:44
  • @Michael I totally wouldn't put it past myself to misdiagnose lol. However, in this case, I put a flow log on the bastion's public subnet, and the SSH traffic that was rejected had the bastion's public IP for the "from" address and the other instance's private IP for the "to" address, hence my diagnosis. However, given my previous comment, now I'm not sure how the 2nd instance's private IP was resolved. Is EC2 just smart enough to figure out private IPs during local traffic? – Rabadash8820 Jan 27 '17 at 03:49
  • @Kfactor21, both subnets use the same route table. It has a row for "local" traffic (10.0.0.0/16) and a row that sends everything else (0.0.0.0/0) to an Internet gateway. I think the issue is probably more related to this DNS situation – Rabadash8820 Jan 27 '17 at 03:52
  • Ah. I can see a case where your observation makes sense but is skewed. Kudos for looking at the flow logs. Note: If the instances on the private subnet have public IPs, that's a misconfiguration (assuming it's a proper private subnet, in which case the default route will not be the Internet Gateway). What's actually happening is the bastion is trying to access the target's *public* IP, which would hairpin through the Internet Gateway, which would translate (static NAT) the target instance's public IP into the target instance's *private* IP, but would leave the bastion's address unchanged. – Michael - sqlbot Jan 27 '17 at 05:28
  • @Rabadash8820 Usually the `/etc/resolv.conf` for an EC2 instance includes Amazon's nameservers, and those nameservers _usually_ resolve the DNS name for an EC2 instance to its _private_ IP address (if resolved from within AWS, of course). This then brings up the question of just _which DNS names_ your bastion host is resolving: are these custom DNS names, or the AWS-assigned DNS names for your EC2 instances? – Castaglia Jan 27 '17 at 06:13
  • @Castaglia I am using custom DNS names, which may be the issue. See my edited post where I included the actual flow logs – Rabadash8820 Jan 27 '17 at 14:23
  • @Michael-sqlbot you're right, my instance is not exactly as "private" as I've been leading on. See the edit in my post, which includes the actual flow logs – Rabadash8820 Jan 27 '17 at 14:24
  • @Rabadash8820 If your custom DNS names are `CNAME` records to the AWS-assigned public DNS hostnames for your instances, _e.g._ "ec2-1-2-3-4...", then it _might_ work properly. The bastion host would resolve your custom DNS name to the instance's public DNS hostname, and then resolve that to the (hopefully) _internal_ IP address. – Castaglia Jan 27 '17 at 15:27
  • @Castaglia Brilliant! At first I was opposed to the idea, b/c I'll have to update the CNAME anytime a new instance with a different IP is used for the web server (which has been happening a lot during this testing phase), but that's really no different than updating the IP in an A record, and either way I'm using a CloudFormation template that should handle all of that automatically every time I update anyway. I'm getting an SSH permission error now but I'm sure that it's unrelated. So yeah, If you want to summarize your above suggestions into some kind of answer, I will totally accept it! – Rabadash8820 Jan 27 '17 at 17:07

2 Answers2

3

After some question/answer back and forth with the OP, we determined that one of the root causes for the issue was the use of custom DNS names.

The user would access the bastion host, and from there would resolve the custom DNS name to a public IP address for the EC2 instance in question. This is why the traffic, from the bastion host, was not using the EC2 instances' private IP addresses. The key to using an EC2 instance's private IP address is to resolve the AWS-assigned public DNS hostname of that instance from within AWS. As the OP noted from the AWS docs, that AWS-assigned public DNS hostname will resolve to a public IP address from outside of AWS, but will resolve to a private IP address from inside of AWS. Thus the key was to get the user using that AWS-assigned public DNS hostname.

One way in which to keep the use of the custom DNS hostname, and still resolve to the private IP address of the EC2 instances from the bastion host is to make the custom DNS name be a CNAME record (rather than an A record), which points to the AWS-assigned public DNS name. Yes, this requires updating that DNS record whenever a new/different EC2 instance appears, but such an update would be required anyway, for an A record to point to the new public IP address. By using a CNAME, things should work as desired.

Hope this helps!

Castaglia
  • 2,972
  • 5
  • 28
  • 49
  • Citation for the split resolution behavior: http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.html. For the benefit of future visitors, note that for this to work, `enableDnsHostnames` and `enableDnsSupport` need to be enabled in the VPC, and you have to be using the AWS-provided resolvers... but there's rarely a case for not doing these things. – Michael - sqlbot Jan 27 '17 at 20:42
0

You need to use CNAME records instead of A records when you setup DNS for your ec2 instance. Assume that you have a www site with url my-site.com. If you put DNS records my-site.com A your_public_ip example my-site.com A 52.14.20.167 your traffic will go outside the aws VPC if you try to reach that url from any server from inside your VPC.

To avoid this you need to change that record to my-site.com CNAME your_EC2_instance_AWS_DNS_address

Example: my-site.com CNAME ec2-XX-YY-ZZZ-17.eu-west-1.compute.amazonaws.com

With this if you access your www service outside AWS you will get your public ip resolved, and inside the AWS VPC you will get the private address of your service and traffic will stay inside AWS VPC.

Minion3665
  • 879
  • 11
  • 25
Aleksandar Lukic
  • 1
  • 1