We have a RedHat 5.8 VM running on ESX on top of Cisco UCS that is acting as a firewall using iptables. The box has multiple nics, one nic serves as the gateway to our network with the other nics each going to a separate VLAN/network. The problem I'm seeing is that doing an iperf test from a host directly to the Firewall VM gives 800+ Mbps speed but trying to do an iperf test from a host outside the firewall to a host inside the firewall (causing the traffic to pass through the firewall) yields speeds of about 30Mbps or less.
The details:
- I'm not sure what release or versions of ESX or UCS we are running, but I can find out if that's important. Our sysadmin keeps it up to date so they're most likely within the last couple of updates.
- I know we are using the VMXNET 3 driver for the network adapters.
- All connections are verified to be 1Gbps.
What I've tried:
- From what I had read, the e1000 driver has better results, so we added a couple of nics using that driver and iperf tested through those interfaces with the same results.
- Checked that LRO was disabled on the tested interfaces. When I ran 'ethtool -K ethX lro off', it reports 'no offload settings changed' which I assume to mean it's already disabled.
- Also disabled TSO on the tested interfaces.
- When I did my speed tests, I was typically testing from a physical device to a virtual device through the firewall on the same cluster. I also tried testing from a virtual device to a virtual device through the firewall and got the same results.
- Disabled iptables and ran speed tests, receiving the same results.
- None of the items above changed anything except possibly making things even slower (I got <10 Mbps at one point).
Because I can get 800+ Mbps going to the firewall itself, that leads me to think that there is nothing wrong with the configuration of the nics themselves. I feel like it's some issue with the forwarding on the OS itself since it only seems to manifest when it's passing traffic through the firewall. I should also note that I didn't observe any CPU spikes during this process.
I'm sure I'm leaving out some details, so if there are any further questions, let me know. I appreciate any help!
