Centos 6.4: Networking issue - Unresponsive network. Cannot reach any external device

Question

I had one server that was in production suddenly become no longer responsive w.r.t network. No settings have been changed in a while, so I don't really understand why it would go wrong. It looks like a routing issue, but don't know how to progress from here.

I tried loading rescue cd and seeing if there's any hardware issue, but temporary OS had no issue connecting via network.

Here are some info: (to note, eth0 is what I use for lan, eth1 is what I use for internet)

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
149.255.34.168  *               255.255.255.248 U     0      0        0 eth1
192.168.1.0     *               255.255.255.0   U     0      0        0 eth0
link-local      *               255.255.0.0     U     1002   0        0 eth0
link-local      *               255.255.0.0     U     1003   0        0 eth1
default         149.255.34.169  0.0.0.0         UG    0      0        0 eth1

Pinging any other devices in vlan/self works fine.

# ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.129 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.193 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.137 ms
...

Other devices in the vlan can connect to this machine as well.

But any via internet is not

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 Destination Host Unreachable
64 bytes from 8.8.8.8: icmp_seq=2 Destination Host Unreachable
64 bytes from 8.8.8.8: icmp_seq=3 Destination Host Unreachable
...

and ifconfig.

# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:25:90:53:9D:D3
          inet addr:192.168.1.4  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::225:90ff:fe50:1abf/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:21 errors:0 dropped:0 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1830 (1.7 KiB)  TX bytes:2868 (2.8 KiB)
          Interrupt:20 Memory:fba00000-fba20000

eth1      Link encap:Ethernet  HWaddr 00:25:90:53:9D:D2
          inet addr:149.255.34.173  Bcast:149.255.34.175  Mask:255.255.255.248
          inet6 addr: fe80::225:90ff:fe50:1abe/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0     <--- nothing ever recieved
          TX packets:730 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:31092 (30.3 KiB)
          Interrupt:16 Memory:fb900000-fb920000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:497 errors:0 dropped:0 overruns:0 frame:0
          TX packets:497 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:54618 (53.3 KiB)  TX bytes:54618 (53.3 KiB)

and it's settings (which is same as another server that's running fine, with exception of assigned ip)

#cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
BOOTPROTO=static
BROADCAST=149.255.34.175
DNS1=8.8.8.8
GATEWAY=149.255.34.169
HWADDR=00:25:90:50:9D:D2
IPADDR=149.255.34.173
IPV6INIT=yes
IPV6_AUTOCONF=yes
NETMASK=255.255.255.248
NM_CONTROLLED=yes
ONBOOT=yes
TYPE=Ethernet
UUID=*************

Edit for more info

# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=173.34.255.149
GATEWAY=149.255.34.169

I cannot ping my gateway. It's another Destination Host Unreachable

Unfortunately, traceroute (or similar) is not installed on the server right now and cannot install due to absence of network.

I have checked with iptables as off.

Can you post your `/etc/sysconfig/network` and a traceroute/tracepath to some external IP (like google's DNS)? Do you define your default gateway in an interface instead of `/etc/sysconfig/network`? Can you ping your default gateway? — dawud, May 13 '13 at 06:33
@dawud, I've edited my original post to include more information. — Grumpy, May 13 '13 at 06:45
Can you ping any other IP's in the subnet? i.e. 149.255.34.170, 149.255.34.171, 149.255.34.172, 149.255.34.174? — Jaz005, May 13 '13 at 07:04
@Jaz005, any address that's not 192.* or localhost will result in `Destination Host Unreachable`. — Grumpy, May 13 '13 at 07:11
This route is useless: "149.255.34.168 * 255.255.255.248 U 0 0 0 eth1", default gateway is enough for this — dawud, May 13 '13 at 07:23
Since you define your default gateway in `/etc/sysconfig/network` instead of the interface, have you tried removing the GATEWAY line from ifcfg-eth1 and restart the network service? — dawud, May 13 '13 at 07:36
@dawud I just tried that per your suggestion but didn't show any change. — Grumpy, May 13 '13 at 07:50
If you still can't ping your gateway on the same network, this is probably a layer 2 issue. Is this a physical or virtual host? if the former, contact the people administering the network to check if your MAC appears in the switch port is should; if the latter, the same applies, and also check if the udev rules are OK — dawud, May 13 '13 at 08:26

score 1 · Accepted Answer · answered Jun 02 '13 at 09:13

Just to bring closure to this.

Though other sys admins have looked at the server, no one was able to find any software related issue. I suspect it was an ethernet adapter failure and so does the host. A brand new server replaced it to come to a solution.

score 0 · Answer 2 · answered May 16 '13 at 06:41

Your HWADDR in /etc/sysconfig/network-scripts/ifcfg-eth1 is wrong.

#cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
HWADDR=00:25:90:50:9D:D2

and your ifconfig eth1 output is

eth1      Link encap:Ethernet  HWaddr 00:25:90:53:9D:D2
          inet addr:149.255.34.173  Bcast:149.255.34.175  Mask:255.255.255.248
          inet6 addr: fe80::225:90ff:fe50:1abe/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0     <--- nothing ever recieved
          TX packets:730 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:31092 (30.3 KiB)
          Interrupt:16 Memory:fb900000-fb920000

Correct the HWADDR and let's see if this fixed your problem.

Centos 6.4: Networking issue - Unresponsive network. Cannot reach any external device

2 Answers2