How to achieve gigabit speeds with Ubuntu using bonding?

Question

I'm working with the bonding driver on Linux (Ubuntu 12.04.2 LTS).

I have 2 PCs with 2 NICs (gigabit ethernet - Intel 82574L) in each one (in total, 4 NICs). I want to improve my TCP throughput over a gigabit network, using bonding, but my problem is that I can't achieve gigabit speeds using bonding.

My gigabit interface can reach 370 - 375 Mbps using a single gigabit interface (without using bonding). When I turn on my bonding, using mode 0 (balance-rr), for example, I can reach 760 - 772 Mbps (using iperf).

When I want to transfer data from one PC to another (using bonding mode balance-rr), I just can reach 472 Mbps (59 MB/s). To tranfer data, I use scp.

My questions are:

What are the limitations of gigabit ethernet over Linux?
Why can't I achieve gigabit speeds?
Can anyone help me to change the default settings on Ubuntu to achieve gigabit speeds?

Can you advise on how to tune Ubuntu for achieve gigabit speeds?

My sysctl.conf:

# cat sysctl.conf | grep -v ^# | tr -s '\n' 
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1


# ethtool -k eth1
Offload parameters for eth1:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off


# ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: off


# ethtool -g eth1
Ring parameters for eth1:
Pre-set maximums:
RX:        4096
RX Mini:    0
RX Jumbo:    0
TX:        4096
Current hardware settings:
RX:        256
RX Mini:    0
RX Jumbo:    0
TX:        256


# ethtool -g eth2
Ring parameters for eth2:
Pre-set maximums:
RX:        4096
RX Mini:    0
RX Jumbo:    0
TX:        4096
Current hardware settings:
RX:        256
RX Mini:    0
RX Jumbo:    0
TX:        256

You could potentially be maxing out on your disks. How are your disks configured? — Nathan C, Jun 07 '13 at 16:09
Or possibly the CPU if you are using an encrypted transfer protocol. You should test with iperf. — Zoredache, Jun 07 '13 at 18:32
+1 - Even on fast machines there is a substantial penalty for scp vs just using netcat or even FTP. — rnxrx, Jun 08 '13 at 02:36

score 2 · Answer 1 · answered Jun 07 '13 at 20:16

2

A single stream can only ever be received on a single link. So while your links may be bonded and the packets split in a round robin fashion on the way out, they are still received by just a single link on the other side.

answered Jun 07 '13 at 20:16

longneck

23,082
4
52
86

How to achieve gigabit speeds with Ubuntu using bonding?

1 Answers1