0

I have two windows Serer 2016 web servers in the public subnets:

  1. Each web server have their own public IP addresses.
  2. Each web server has self-signed SSL certificate, has both HTTP and HTTPS bindings, and redirecting HTTP to HTTPS.
  3. They are in a Target Group. There is an Application Load Balancer using that Target Group.

I could request the pages from the web servers directly or from the ELBs. Response is instant. All good.

Once I removed the public IP addresses from the web servers, the response from the ELB becomes very unreliable. It times out every now and then with error 504 Gateway Timeout. But sometimes it does work.

Why?

Silly Dude
  • 558
  • 3
  • 9
  • 22
  • is your app making any outbound traffic when you make a request to it other then returning the response? – Mike Sep 13 '17 at 00:31
  • The web servers talk to the MySQL database in private subnet. No other outbound traffic. – Silly Dude Sep 13 '17 at 00:32
  • Is the ELB communicating with the instances securely over TCP/443? If so, how is name resolution configured? –  Sep 13 '17 at 01:18
  • The ELB allows both HTTP and HTTPS, but the web servers redirects HTTP to HTTPS. I am not sure about the name resolution. I thought the ELB knows about that target group, and it distributes traffic evenly across all instances in that target group. That is all I know. – Silly Dude Sep 13 '17 at 01:21
  • 1
    my only guess is you have traffic leaking out somewhere. Connections starting from the public subnet outside the VPC will not work without a public IP. I'd run tcpdump and make some connections to see if you are making outside connections. – Mike Sep 13 '17 at 02:11
  • Does your application involve the client connecting to different domains that are hosted on your servers? Maybe you didn't update all the necessary DNS records so that all domains / hosts point to the ELB. – Appleoddity Sep 13 '17 at 03:50
  • Also have you enabled logging and reviewed the logs and health of your ELB? – Appleoddity Sep 13 '17 at 03:50
  • I ran perfmon on the server, and found that the IIS was making two types of outside connections, one is to AWS S3, which is expected, the other is to Microsoft (52.243.81.129, 52.231.249.9), which I don't understand why. – Silly Dude Sep 17 '17 at 23:25

2 Answers2

1

A cheaper solution than a NAT gateway is an S3 Endpoint. From that page

The source IPv4 addresses from instances in your affected subnets as received by Amazon S3 will change from public IPv4 addresses to the private IPv4 addresses from your VPC. An endpoint switches network routes, and disconnects open TCP connections. Your tasks will be interrupted during the changeover, and any previous connections using public IPv4 addresses will not be resumed.

This means you don't have to pay for a NAT gateway, just traffic to S3. Note that

Endpoints currently do not support cross-region requests

Tim
  • 31,888
  • 7
  • 52
  • 78
0

With the help of Mike, Ben and Appleoddity I finally figured out. The web servers does need traffic to outside because they are access AWS S3. They stop functioning as soon as their public IPs are removed, because they need it to connect to the Internet. The proper implementation that maximums the security is to place the web servers in the private subnets, and add a NAT gateway to each corresponding public subnet, and add a route in the route table to route traffic to 0.0.0.0/0 to the NAT gateway.

Silly Dude
  • 558
  • 3
  • 9
  • 22