In the AWS Network Load Balancer documentation it says that when specifying instances for a Target Group that it must include an instance in every AZ that the Load Balancer is registered in. This is not enforced.
What happens to traffic if you have an NLB registered in 3 AZs, but only a single target EC2 instance in AZ1? What if you enable cross AZ load balancing, does that make any difference?
Your scenario wouldn't require cross-zone load balancing be enabled. As Marcin pointed it, it does nothing for you. In fact, resolve your NLB's DNS and you'll see that it only returns an A record for each AZ that has a healthy instance aggregated in all of the NLB's target gouups. Marcin's response is fantastic for the deep dive.
Some folks are here saying "yes, but we're still getting timeouts.". This is because your scenario is more complex that OP's. In short, you likely have an NLB with more than one target group in which distinct targets exist in multiple AZs. Turning on cross-zone load balancing will solve your suboptimal configuration at the cost of additional data transfer charges on your bill. Duck tape and gum wrappers work in AWS VPC.
More info:
NLBs are "smart" with DNS in that their VIP will only resolve to A records that have healthy targets (within a timely reason). If an NLB has several target groups with distinct instances across three AZs, you'll be getting three A records returned (one for each AZ that has a healthy target). This is how NLB + RR DNS works.
However, if your target groups contains EC2 instance(s) in a single AZ then it's a 33% (given three AZs) chance that the DNS round robin will resolve the proper AZ.
The best solution is to turn on cross-zone load balancing. This increases data transfer cost, but it is less complex to the alternative of breaking out the NLBs. Please note that enabling cross-zone load balancing will take a few minutes to kick in. Don't enable it, kick off a telnet immediately and be sad when it doesn't work. Wait 5 - 10 minutes and then kick off your telnet.
Source: Anecdotal and practical experience working with AWS and janky EC2 solutions.
What happens to traffic if you have an NLB registered in 3 AZs, but only a single target EC2 instance in AZ1? What if you enable cross AZ load balancing, does that make any difference?
In this particular scenario (NLB in 3 AZs, and single instance in 1 AZ), nothing really happens. There is no apparent difference with, or without, cross-zone load balancing from the perspective of the end-user. The instance will be accessible in either case.
To verity that, I developed a simple CloudFormation template the creates NLB, with, or without cross-zone load balancing, and 1 instance. The template allows for easy experimentation with different setups of NLB, cross-zone and instance location. I used the template in us-east-1 region and default VPC.
For the template you specify several parameters, including:
NLBSubnetsIds - subnets where to enable NLB. You have to check first in console, which subnets are in which AZs.
InstanceSubnetId - subnet for the instance. Again you can check which subnet is in which AZ if you want play around with instance location. You must ensure that instance is created in one of AZs set for your NLB.
CrossZoneEnabled - enable or disable cross-zone balancing for the NLB.
Once you create the stack from the template and instance health check pass (can take 1 or 2 minutes), you can access NLB DNS in your browser to view a sample webpage hosted on the instance.
---
Parameters:
VpcId:
Type: AWS::EC2::VPC::Id
NLBSubnetsIds:
Type: List<AWS::EC2::Subnet::Id>
InstanceSubnetId:
Type: AWS::EC2::Subnet::Id
AmazonLinux2AMIId:
Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
CrossZoneEnabled:
Type: String
Default: false
AllowedValues: [true, false]
Resources:
BasicSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Enable www port
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 80
ToPort: 80
CidrIp: 0.0.0.0/0
VpcId: !Ref VpcId
MyInstance1:
Type: AWS::EC2::Instance
CreationPolicy:
ResourceSignal:
Timeout: PT5M
Properties:
ImageId: !Ref AmazonLinux2AMIId
InstanceType: t2.micro
Monitoring: false
SecurityGroupIds: [!Ref BasicSecurityGroup]
SubnetId: !Ref InstanceSubnetId
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
yum install -y httpd aws-cfn-bootstrap
echo "<h2>Hello world from $(hostname -f)</h2>" \
> /var/www/html/index.html
systemctl start httpd
# check if website is working
curl -s localhost | grep "Hello"
# Signal the status from cfn-init
/opt/aws/bin/cfn-signal -e $? \
--stack ${AWS::StackName} \
--resource MyInstance1 \
--region ${AWS::Region}
MyNLB:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
IpAddressType: ipv4
LoadBalancerAttributes:
- Key: load_balancing.cross_zone.enabled
Value: !Ref CrossZoneEnabled
Scheme: internet-facing
Subnets: !Ref NLBSubnetsIds
Type: network
MyListner1:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- TargetGroupArn: !Ref MyTargetGroup
Type: forward
LoadBalancerArn: !Ref MyNLB
Port: 80
Protocol: TCP
MyTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
HealthCheckEnabled: true
HealthCheckIntervalSeconds: 10
HealthCheckPath: /
HealthCheckProtocol: HTTP
HealthyThresholdCount: 2
UnhealthyThresholdCount: 2
Port: 80
Protocol: TCP
TargetGroupAttributes:
- Key: deregistration_delay.timeout_seconds
Value: 30
Targets:
- Id: !Ref MyInstance1
Port: 80
TargetType: instance
VpcId: !Ref VpcId
Outputs:
DNSName:
Value: !GetAtt MyNLB.DNSName
From the end-user perspective, in your scenario there is no clear difference between enabling or disabling cross-zone in NLB. However, the long-term difference could be in high availability. Namely, if you have cross-zone disabled and if something happens with a NLB node in the AZ where the instance is located, NLB won't be able to route traffic to your instance from other AZ. This is my speculation, as this is not something which you can check manually. The reason is that once you associate an AZ/subnet with your NLB, you can't disassociate it, to check what happens in such scenario.
In contrast, if cross-zone is enabled, in the above scenario, NLB node from other zone could probably route traffic to the instance across zones.
The major benefit of having cross-zone traffic enabled, is when you different number of instances in different AZs. In this case, cross-zone balancing enables that all instances will get roughly same amount of traffic. Without, cross-zone balancing, an isolate instance would get much more traffic then the collection of instances in the other AZ.
You can check the effects of zone-balancing using the second template. The template almost same as before, but now 1 AZ will have 3 instances, while the other one will have 1 AZ.
---
Parameters:
VpcId:
Type: AWS::EC2::VPC::Id
NLBSubnetsIds:
Type: List<AWS::EC2::Subnet::Id>
InstanceSubnetId1:
Type: AWS::EC2::Subnet::Id
InstanceSubnetId2:
Type: AWS::EC2::Subnet::Id
AmazonLinux2AMIId:
Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
Default: /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2
CrossZoneEnabled:
Type: String
Default: false
AllowedValues: [true, false]
Resources:
BasicSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Enable www port
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: 80
ToPort: 80
CidrIp: 0.0.0.0/0
VpcId: !Ref VpcId
MyInstance1:
Type: AWS::EC2::Instance
CreationPolicy:
ResourceSignal:
Timeout: PT3M
Properties:
ImageId: !Ref AmazonLinux2AMIId
InstanceType: t2.micro
Monitoring: false
SecurityGroupIds: [!Ref BasicSecurityGroup]
SubnetId: !Ref InstanceSubnetId1
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
yum install -y httpd aws-cfn-bootstrap
echo "<h2>Hello world from $(hostname -f)</h2>" \
> /var/www/html/index.html
systemctl start httpd
# check if website is working
curl -s localhost | grep "Hello"
# Signal the status from cfn-init
/opt/aws/bin/cfn-signal -e $? \
--stack ${AWS::StackName} \
--resource MyInstance1 \
--region ${AWS::Region}
MyInstance2:
Type: AWS::EC2::Instance
CreationPolicy:
ResourceSignal:
Timeout: PT3M
Properties:
ImageId: !Ref AmazonLinux2AMIId
InstanceType: t2.micro
Monitoring: false
SecurityGroupIds: [!Ref BasicSecurityGroup]
SubnetId: !Ref InstanceSubnetId2
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
yum install -y httpd aws-cfn-bootstrap
echo "<h2>Hello2 world from $(hostname -f)</h2>" \
> /var/www/html/index.html
systemctl start httpd
# check if website is working
curl -s localhost | grep "Hello"
# Signal the status from cfn-init
/opt/aws/bin/cfn-signal -e $? \
--stack ${AWS::StackName} \
--resource MyInstance2 \
--region ${AWS::Region}
MyInstance3:
Type: AWS::EC2::Instance
CreationPolicy:
ResourceSignal:
Timeout: PT3M
Properties:
ImageId: !Ref AmazonLinux2AMIId
InstanceType: t2.micro
Monitoring: false
SecurityGroupIds: [!Ref BasicSecurityGroup]
SubnetId: !Ref InstanceSubnetId2
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
yum install -y httpd aws-cfn-bootstrap
echo "<h2>Hello2 world from $(hostname -f)</h2>" \
> /var/www/html/index.html
systemctl start httpd
# check if website is working
curl -s localhost | grep "Hello"
# Signal the status from cfn-init
/opt/aws/bin/cfn-signal -e $? \
--stack ${AWS::StackName} \
--resource MyInstance3 \
--region ${AWS::Region}
MyInstance4:
Type: AWS::EC2::Instance
CreationPolicy:
ResourceSignal:
Timeout: PT3M
Properties:
ImageId: !Ref AmazonLinux2AMIId
InstanceType: t2.micro
Monitoring: false
SecurityGroupIds: [!Ref BasicSecurityGroup]
SubnetId: !Ref InstanceSubnetId2
UserData:
Fn::Base64: !Sub |
#!/bin/bash -xe
yum install -y httpd aws-cfn-bootstrap
echo "<h2>Hello2 world from $(hostname -f)</h2>" \
> /var/www/html/index.html
systemctl start httpd
# check if website is working
curl -s localhost | grep "Hello"
# Signal the status from cfn-init
/opt/aws/bin/cfn-signal -e $? \
--stack ${AWS::StackName} \
--resource MyInstance4 \
--region ${AWS::Region}
MyNLB:
Type: AWS::ElasticLoadBalancingV2::LoadBalancer
Properties:
IpAddressType: ipv4
LoadBalancerAttributes:
- Key: load_balancing.cross_zone.enabled
Value: !Ref CrossZoneEnabled
Scheme: internet-facing
Subnets: !Ref NLBSubnetsIds
Type: network
MyListner1:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- TargetGroupArn: !Ref MyTargetGroup
Type: forward
LoadBalancerArn: !Ref MyNLB
Port: 80
Protocol: TCP
MyTargetGroup:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
HealthCheckEnabled: true
HealthCheckIntervalSeconds: 10
HealthCheckPath: /
HealthCheckProtocol: HTTP
HealthyThresholdCount: 2
UnhealthyThresholdCount: 2
Port: 80
Protocol: TCP
TargetGroupAttributes:
- Key: deregistration_delay.timeout_seconds
Value: 30
Targets:
- Id: !Ref MyInstance1
Port: 80
- Id: !Ref MyInstance2
Port: 80
- Id: !Ref MyInstance3
Port: 80
- Id: !Ref MyInstance4
Port: 80
TargetType: instance
VpcId: !Ref VpcId
Outputs:
DNSName:
Value: !GetAtt MyNLB.DNSName
If you use the above template, and repeatedly request the NLB url, you will see that the isolated instance will get about 50% of the traffic without cross-zone balancing. With cross-zone balancing enabled, it will be about 20%. Below are my results based on 100 requests:
