1

I recently set up 2 new R610 servers that I purchased.

Summary:

Both servers 100% same hardware Both installed with Centos 6.7 64bit Tested multiple switch ports, same slowness issue. Both same full duplex / 100mbit speed BIOS 6.4.0

One server is fine on test results around 10Mbps. The problematic server is constantly slow, less than 1MBps. It's obviously noticeable just logging in via ssh as well.

[root@server ~]# wget cachefly.cachefly.net/100mb.test
--2015-11-16 12:03:35-- cachefly.cachefly.net/100mb.test
Resolving cachefly.cachefly.net... 205.234.175.175 Connecting to cachefly.cachefly.net|205.234.175.175|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 104857600 (100M) [application/octet-stream] Saving to: “100mb.testâ€

6% [============>                                                                                                                                                                                  ] 7,233,814   538K/s eta 2m 56s

Same switch configuration

Fa0/1                       connected   9            full   100 10/100BaseTX                 (good server)
Fa0/20                       connected   10           full   100 10/100BaseTX              (slow server)[

It was replicated on another switch port as well. Same problem. Kind of odd since both server's hardware are exactly the same. I'm thinking it's a setting within the BIOS, but I checked and they both look exact.

Installed latest NIC firmware for BCM5709, rebooted. Still slow

[root@server ~]# ethtool -i em1
driver: bnx2
version: 2.2.5
firmware-version: 7.10.18 bc 7.10.0 NCSI 2.0.13
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

[root@server ~]# ethtool em1
Settings for em1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full 
                                100baseT/Half 100baseT/Full 
                                1000baseT/Full 
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: 100Mb/s
        Duplex: Half
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: off
        Supports Wake-on: g
        Wake-on: d
        Link detected: yes


[root@server ~]# dmesg | grep -i duplex
bnx2 0000:01:00.0: em1: NIC Copper Link is Up, 100 Mbps half duplex
Cazzette
  • 63
  • 1
  • 11
  • Your question says "full duplex", but the output of your ethtool says "half duplex", I believe you have a duplex mismatch between your server and the switch. – oo. Nov 17 '15 at 01:11
  • @oo the weird thing is, the other server says half duplex and works fine, consistently at 10Mbps. Anyways, how can I correct this to full duplex? – Cazzette Nov 17 '15 at 01:30
  • Have you tried swapping cables? (after 1 bad cable I got them to replace all cables with high quality manufactured cables, threw away the cables they made themselves) Another old NIC trick which sometimes helped crazy NIC cards was power off and pull power out and leave powered off for a few mins. – jqa Nov 17 '15 at 02:47
  • @james forgot to mention that I replaced the new cables with another set of new cables. When you say power off and pull power out, what's the difference compared to just powering down via DRAC and waiting a few minutes to power on? I just racked this server up yesterday too. – Cazzette Nov 17 '15 at 03:16
  • @Cazzette - learned the power draining from a admin years ago. After trying every possible config and nothing works, i re-config it all back to what it "should" work, pull the power and let it drain. What I was told was the NICs have some of their own memory (so they can WLAN ??) Maybe it is just a tech placebo – jqa Nov 17 '15 at 16:45

1 Answers1

0

I think you should be receiving more than 10Mbps. Try: ethtool -s em1 speed 100 duplex full autoneg off.

If that works, then make the change perms to by adding it to the em1 configuration file by way of this: http://www.cyberciti.biz/tips/howto-linux-add-ethtool-duplex-settings-permanent.html

oo.
  • 861
  • 6
  • 11
  • Unfortunately that did not solve the issue with the bad server. The other server was getting consistent 11.3Mb/s on cachefly. But the slow one, about 200K/s. AI did a service network restart, tried again. Same thing. @oo – Cazzette Nov 17 '15 at 01:43
  • The "service network restart" will undo your ethtool manual setting of duplex. After changing the duplex on the server side, did you verify that ethtool showed the system at full duplex? Did you also check the switch to make sure it's also showing full duplex and has auto negotiation disabled? – oo. Nov 17 '15 at 01:55
  • I re-ran that command and checked: it shows full on switch and server, but still slow. Duplex: Full, Speed: 100Mb/s.@oo – Cazzette Nov 17 '15 at 01:58
  • Is em1 the active outbound interface? Does your default route point to it? – oo. Nov 17 '15 at 01:59
  • Yes, em1 is default outbound. http://pastebin.com/KCjX2uXy @oo – Cazzette Nov 17 '15 at 02:02
  • Unfortunately, there's not a whole lot more I can add -- you'll need to do some testing (using a tool such as iperf), to track down the root cause. But I can say for certain that speed/duplex mismatches definitely do not help your network speed. – oo. Nov 17 '15 at 02:04
  • what about this? 3 lines, one is half and others full? http://pastebin.com/hEXM179P – Cazzette Nov 17 '15 at 02:11
  • I believe those log entries are from the two times you ran ethtool. If "ethtool em1" shows full duplex, then you're at full duplex. – oo. Nov 17 '15 at 02:12