The same driver for multiple network cards -- performance bottleneck?

Question

I'm using driver e1000e for multiple Intel network cards (Intel EXPI9402PT, based on 82571EB chip). The problem is that when I'm trying to utilize maximum speed (1GB) on more than one interface, speed on each interface starts to drop down.

For one interface I get: 120435948 bytes/sec.

For two interfaces I get: 61080233 bytes/sec and 60515294 bytes/sec.

For three interfaces I get: 28564020 bytes/sec, 27111184 bytes/sec, 27118907 bytes/sec.

What can be the cause?

EDIT: /proc/interrupts content:

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
106:      17138          0          0          0          0          0          0          0         PCI-MSI  eth0
114:         51          0          0          0     102193          0         20   23745467         PCI-MSI  eth2
122:         51        290         15        271          0       9253        100          0         PCI-MSI  eth3
130:         43        367          0        290        105         39         15          0         PCI-MSI  eth4
138:         43        361        105        210          0        140          0          0         PCI-MSI  eth5
146:         56      67625        100          0          0   17855245          0          0         PCI-MSI  eth6

Are these interfaces bonded or independent? If they are PCI, are you using a 64bit OS? — Andy, Jun 18 '09 at 11:08
@Andy: two interfaces are on one card connected to PCI-Express x4. Another two are on another card connected to PCI-Express x8. It's 64 bit OS. — ctinnist, Jun 18 '09 at 11:22

Dan Carley · Answer 1 · 2009-06-18T11:32:20.857

6

It won't be the driver.

It's most likely to be a physically shared component, such as interrupts or the PCI bus.

edited Jun 18 '09 at 11:32

answered Jun 18 '09 at 11:02

Dan Carley

25,617
5
53
70

The bus is PCI-Express x4 and x8. – ctinnist Jun 18 '09 at 11:21
but are they on the SAME bus and do they share interrupts? – Chopper3 Jun 18 '09 at 12:14

score 2 · Answer 2 · answered Jun 18 '09 at 11:22

2

Are they sharing the same interrupt (IRQ)? This is probably your bottleneck.

answered Jun 18 '09 at 11:22

pauska

19,620
5
57
75

3

dstat is quite a nice iostat-like utility which will allow you to view the interrupt count, CPU toll and other stats. Try running it at the same time as an iperf test. – Dan Carley Jun 18 '09 at 11:37
I checked /proc/interrupts, and it looks like every interface has its own interrupt (i.e. each interface is in distinct line). – ctinnist Jun 18 '09 at 11:53
Hmm. Is this a SMP server? Debian or Ubuntu? – pauska Jun 18 '09 at 12:03
Yes, it's SMP, 8 cores, CentOS – ctinnist Jun 18 '09 at 12:24
Ok. Try to monitor /proc/interrupts while doing I/O on all network ports simultaniously. The interrupts should even out across the CPU's. – pauska Jun 18 '09 at 15:09

score 2 · Answer 3 · answered Jun 18 '09 at 13:30

2

What is the endpoint of your iperf test? If you are routing through network hardware or combining all output to a single GBe NIC on another machine your bottleneck may be remote.

answered Jun 18 '09 at 13:30

Andy

5,230
1
24
34

score 1 · Answer 4 · edited Apr 13 '17 at 12:14

1

I've posted some sysctl magic here. You can try it, see if it helps

PS. How you benchmarking speed?

edited Apr 13 '17 at 12:14

Community

1

answered Jun 18 '09 at 13:24

SaveTheRbtz

5,691
4
32
45

I'm benchmarking it by a script that grabs data from /proc/net/dev (twice) and calculates the speed. – ctinnist Jun 18 '09 at 14:28

The same driver for multiple network cards -- performance bottleneck?

4 Answers4