I have 2 servers each with 2 network cards running through two totally separate networks. One network is a public facing network and the NICs are configured with public IP's. The other network is a private network connecting the two servers with private IP's.
Server 1 IPs:
197.189.20x.x / 255.255.255.248 / GW: 197.189.20x.x - Public NIC plugged into public network
10.10.10.1 / 255.255.255.0 / GW: None - Private NIC plugged into private network
Server 2 IPs:
197.189.21x.x / 255.255.255.248 / GW: 197.189.21x.x - Public NIC plugged into public network
10.10.10.3 / 255.255.255.0 / GW: None - Private NIC plugged into private network
Regardless of which IP the servers connect to each other with, the traffic is being sent through the public NIC (which is 100mb), then when it reaches it's throughput limit, it puts the additional traffic on the private NIC (which is 1gb).
To further fuel the confusion, when I disable the public nic, I get a full 1gb transfer, when I enable the public nic and it "load balances itself", then I only get around 300mb (100 from public, 200 from private).
I don't know if this will mean anything, but these two servers were replaced this week. Previously we had two servers running Windows Server 2008 R2 with different makes of on-board NICs; the new servers have been loaded with Windows Server 2012 R2 and are running identical Intel on-board NICs. So the network is the same, it's just the servers and their OS's that are now different. On the two 2008 servers everything was working perfectly out the box, private traffic was private and public traffic was public.
Any help would be much appreciated as I'm ready to give up and go get hammered instead!
Edit: In a desperate measure, I started regressing the servers in an attempt to try isolate the cause of the issue. As soon as I removed the Virtual Switches from Hyper-V, the network immediately started responding as expected. Put the switches back and it starts dynamically load balancing again. That helped narrow my search and I found this article "What's New in Hyper-V Virtual Switch in Windows Server 2012 R2"
Dynamic Load Balancing of Network Traffic
Windows Server® 2012 provided simultaneous load distribution and failover, but did not ensure load distribution between the NICs in a NIC team in a balanced manner.
In Windows Server® 2012 R2, dynamic load balancing continuously and automatically moves traffic streams from NIC to NIC within the NIC team to share the traffic load as equitably as possible.
For more information on NIC Teaming, see NIC Teaming Overview.
The only kicker is, we're NOT USING NIC TEAMING! Hopefully this sheds some light on the subject and here's to hoping someone's encountered and managed to solve this dilemma.
Edit 2: Here is the routing table (Hyper-V Switches have been created again):
Interface List
22...0c c4 7a 71 a9 01 ......Hyper-V Virtual Ethernet Adapter #3
20...0c c4 7a 71 a9 00 ......Hyper-V Virtual Ethernet Adapter #2
1...........................Software Loopback Interface 1
14...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
15...00 00 00 00 00 00 00 e0 Microsoft 6to4 Adapter
16...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
===========================================================================
IPv4 Route Table
===========================================================================
Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 197.189.20x.x2 197.189.20x.x3 261
10.10.10.0 255.255.255.0 On-link 10.10.10.1 261
10.10.10.1 255.255.255.255 On-link 10.10.10.1 261
10.10.10.255 255.255.255.255 On-link 10.10.10.1 261
127.0.0.0 255.0.0.0 On-link 127.0.0.1 306
127.0.0.1 255.255.255.255 On-link 127.0.0.1 306
127.255.255.255 255.255.255.255 On-link 127.0.0.1 306
197.189.20x.x1 255.255.255.248 On-link 197.189.20x.x3 261
197.189.20x.x3 255.255.255.255 On-link 197.189.20x.x3 261
197.189.20x.x4 255.255.255.255 On-link 197.189.20x.x3 261
224.0.0.0 240.0.0.0 On-link 127.0.0.1 306
224.0.0.0 240.0.0.0 On-link 197.189.20x.x3 261
224.0.0.0 240.0.0.0 On-link 10.10.10.1 261
255.255.255.255 255.255.255.255 On-link 127.0.0.1 306
255.255.255.255 255.255.255.255 On-link 197.189.20x.x3 261
255.255.255.255 255.255.255.255 On-link 10.10.10.1 261
===========================================================================
Persistent Routes:
Network Address Netmask Gateway Address Metric
0.0.0.0 0.0.0.0 197.189.20x.x2 Default
0.0.0.0 0.0.0.0 197.189.20x.x2 256
===========================================================================
IPv6 Route Table
===========================================================================
Active Routes:
If Metric Network Destination Gateway
15 1177 ::/0 2002:c058:6301::1
15 1176 ::/0 2002:c058:6301::c058:6301
1 306 ::1/128 On-link
15 1005 2002::/16 On-link
15 261 2002:c5bd:cb0a::c5bd:cb0a/128
On-link
1 306 ff00::/8 On-link
===========================================================================
Persistent Routes:
None
VSwitch configs for both servers are identical:
Private network is an "External Network" switch connected to the NIC running the private 10.x IPs.
Public network is an "External Network" switch connected to the NIC running public 197.x IPs.
No VLANs configured
Appreciate the assist, let me know if you need anything else, I'll be here all night :(
