2

I'm using the Intel 82599EB 10G NIC on a server running RHEL6. I ran a program where packets are generated at high rates. The NIC was previously installed on another PC where I was able to generate traffic at 8-9Gbps on each port concurrently. On this server, I was able to generate at 8-9Gbps only if one port is in use. If I were to generate on both ports concurrently, then my total transmit rate drops to about 12Gbps (~6Gbps on each port).

My relevant "lspci -vvv" output shows the following:

0b:00.0 Ethernet controller: ...
      Capabilities: [a0] Express (v2) Endpoint, MSI 00
            LnkCap: Port #1, Speed 5GT/s, Width x8
            LnkSta: Speed 5GT/s, Width x4
            LnkCtl2: Target Link Speed: 5GT/s

0b:00.1 Ethernet controller: ...
      Capabilities: [a0] Express (v2) Endpoint, MSI 00
            LnkCap: Port #1, Speed 5GT/s, Width x8
            LnkSta: Speed 5GT/s, Width x4
            LnkCtl2: Target Link Speed: 2.5GT/s

Is my transmit rate limited by the "x4" in LnkSta for both ports? And what does it mean to have "Target Link Speed: 5GT/s" on one port and "Target Link Speed: 2.5GT/s" on the other port? Does this contribute to the rate decrease too?

Also, I see a couple of PCIE slots that are longer than my NIC's PCIE connectors. Can I install my NIC on these longer slots instead?

Rayne
  • 211
  • 2
  • 14

1 Answers1

3

Also, I see a couple of PCIE slots that are longer than my NIC's PCIE connectors. Can I install my NIC on these longer slots instead?

MOSTLY yes. As in: Yes in theory, not sure in practice - a bad BIOS may blow that. FIor example in the past the Graphics port (a ncice pCIEx16) with a RAI card may result in boot failures due to the BIOS Not handling that well... THAT though is years in the past, these days such problems should not happen. I do that regularly, running RAID controllers from Graphics slots and using on board graphics for server scenarios.

Try and see waht hapens is the only solution here - PCIe is compatible in that CAN put a smaller connector into a larger slot from the standard. The larger connector must not even connect all pins - you could have a PCIex1 link on a PCIex16 connector, advantage clearly that all cards fit but only work with PCIEx1... I have some convertrs here to run graphics cards on PCIEx1 slots for number crunching.

Is my transmit rate limited by the "x4" in LnkSta for both ports?

Well, the physical limit of the port obviously limits the speed of the line, or? And yes, that is a physical limit - nothing you can program around.

And what does it mean to have "Target Link Speed: 5GT/s" on one port and "Target Link Speed: 2.5GT/s" on the other port? Does this contribute to the rate decrease too?

Someone else will have to answer what, but I found this in google:

http://www.xilinx.com/support/answers/37497.htm

If that is as I read it, that is the configured uppe limit - so yes, that owuld be a THE Reason, but the question is why, so it may be a symptom of something else.

TomTom
  • 51,649
  • 7
  • 54
  • 136
  • Re: "Target Link Speed" PCIe is build of of a number of lanes. Each lane of PCIe version 1 could transfer 2.5Gbit/second. PCIe version 2 raised this to 5.0GBit/sec by doubling the speed, and PCIe v3 raised it to 8Gbit/sec by raising speed even more and using a different encoding. It is not rare to have a motherboard which uses both lanes with v1 speed and v2 speeds from different components (e.g. faster lanes coming from a modern CPU, slower lanes from an older chipset). – Hennes Jun 04 '12 at 08:28
  • The LnkCap says both ports are "Width x8", while LnkSta says both ports are currently x4. Does that mean both ports should be capable of better performance if they were using a x8 slot? Are x8 slots the longer slots, and my best option now is to install the NIC in one of these longer slots and see if rates increase? – Rayne Jun 04 '12 at 10:22