We have a DELL R610 server connected to a DELL PowerConnect 5424 switch. This switch is connected to a DELL Equallogic SAN. The DELL R610 serves as a MySQL Database server, the SAN provides the data data directory, mounted as iSCSI drive.
According to an earlier question asked by me here, we are observing that after reboot, MySQL fails to come up on its own. Also this is an intermittent behavior. On investigating we found that iSCSI initiator service fails to execute commands on startup. On examining logs, we found that there is a small blip when our network interfaces come up. Grepped the bnx2 logs:
bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.2.4 (Aug 05, 2013)
bnx2 0000:01:00.0 eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem d6000000, IRQ 36, node addr 5c:f9:dd:f1:8a:ea
bnx2 0000:01:00.1 eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem d8000000, IRQ 48, node addr 5c:f9:dd:f1:8a:ec
bnx2 0000:02:00.0 eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem da000000, IRQ 32, node addr 5c:f9:dd:f1:8a:ee
bnx2 0000:02:00.1 eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem dc000000, IRQ 42, node addr 5c:f9:dd:f1:8a:f0
bnx2 0000:02:00.0: irq 78 for MSI/MSI-X
bnx2 0000:02:00.0: irq 79 for MSI/MSI-X
bnx2 0000:02:00.0: irq 80 for MSI/MSI-X
bnx2 0000:02:00.0: irq 81 for MSI/MSI-X
bnx2 0000:02:00.0: irq 82 for MSI/MSI-X
bnx2 0000:02:00.0: irq 83 for MSI/MSI-X
bnx2 0000:02:00.0: irq 84 for MSI/MSI-X
bnx2 0000:02:00.0: irq 85 for MSI/MSI-X
bnx2 0000:02:00.0: irq 86 for MSI/MSI-X
bnx2 0000:02:00.0 em3: using MSIX
bnx2 0000:01:00.0: irq 87 for MSI/MSI-X
bnx2 0000:01:00.0: irq 88 for MSI/MSI-X
bnx2 0000:01:00.0: irq 89 for MSI/MSI-X
bnx2 0000:01:00.0: irq 90 for MSI/MSI-X
bnx2 0000:01:00.0: irq 91 for MSI/MSI-X
bnx2 0000:01:00.0: irq 92 for MSI/MSI-X
bnx2 0000:01:00.0: irq 93 for MSI/MSI-X
bnx2 0000:01:00.0: irq 94 for MSI/MSI-X
bnx2 0000:01:00.0: irq 95 for MSI/MSI-X
bnx2 0000:01:00.0 em1: using MSIX
bnx2 0000:01:00.1: irq 96 for MSI/MSI-X
bnx2 0000:01:00.1: irq 97 for MSI/MSI-X
bnx2 0000:01:00.1: irq 98 for MSI/MSI-X
bnx2 0000:01:00.1: irq 99 for MSI/MSI-X
bnx2 0000:01:00.1: irq 100 for MSI/MSI-X
bnx2 0000:01:00.1: irq 101 for MSI/MSI-X
bnx2 0000:01:00.1: irq 102 for MSI/MSI-X
bnx2 0000:01:00.1: irq 103 for MSI/MSI-X
bnx2 0000:01:00.1: irq 104 for MSI/MSI-X
bnx2 0000:01:00.1 em2: using MSIX
bnx2 0000:01:00.0 em1: NIC Copper Link is Up, 100 Mbps full duplex
bnx2 0000:02:00.0 em3: NIC Copper Link is Up, 1000 Mbps full duplex
bnx2 0000:01:00.1 em2: NIC Copper Link is Up, 1000 Mbps full duplex
**bnx2 0000:01:00.1 em2: NIC Copper Link is Down**
bnx2 0000:01:00.1 em2: NIC Copper Link is Up, 1000 Mbps full duplex
Our current workaround is to restart the server. So far, on restarting the server, everything comes up smoothly and the blip as shown in the logs above is not observed.
Can anyone help in how to go ahead and troubleshoot what is failing? I have referred here but this is most likely not my case since our issue only occurs at reboot. Other than that there are no NIC errors in ifconfig and there is no packet drop / loss in the NICs. We have never run into any network issue once the server is up.
The DELL R610 is running Ubuntu 14.04.
Some more logs as suggested by @Dom:
$ cat logfile.mysql.withoutdate |grep -B 15 -A 15 "NIC Copper Link is Down"
Loading iSCSI transport class v2.0-870.
bnx2 0000:01:00.1 em2: using MSIX
IPv6: ADDRCONF(NETDEV_UP): em2: link is not ready
iscsi: registered transport (tcp)
iscsi: registered transport (iser)
multipathd (2470): /proc/2470/oom_adj is deprecated, please use /proc/2470/oom_score_adj instead.
bnx2 0000:01:00.0 em1: NIC Copper Link is Up, 100 Mbps full duplex
IPv6: ADDRCONF(NETDEV_CHANGE): em1: link becomes ready
bnx2 0000:02:00.0 em3: NIC Copper Link is Up, 1000 Mbps full duplex
, receive & transmit flow control ON
IPv6: ADDRCONF(NETDEV_CHANGE): em3: link becomes ready
bnx2 0000:01:00.1 em2: NIC Copper Link is Up, 1000 Mbps full duplex
, receive & transmit flow control ON
IPv6: ADDRCONF(NETDEV_CHANGE): em2: link becomes ready
bnx2 0000:01:00.1 em2: NIC Copper Link is Down
bnx2 0000:01:00.1 em2: NIC Copper Link is Up, 1000 Mbps full duplex
, receive & transmit flow control ON
