0

No idea how/why, but here's what I'm seeing. Consider 3 CentOS 5.10 boxes:

Host A  [10.1.2.7]
Host B  [10.1.2.8]
Host C  [10.1.2.9]

All of a sudden we're getting strange behavior from Host X, 10.1.2.3. Upon investigation and a bunch of telnetting to various ports, correlating with host logs, etc., we finally surmise that every host (A, B, C) "thinks" it is 10.1.2.3 as well, such that:

hostA# telnet 10.1.2.3 80

results in Host A itself receiving that TCP/IP connection (port 80 in this example) instead of connecting to the true 10.1.2.3 (Host X). Host A is not configured as such:

hostA# ifconfig -a

eth0      Link encap:Ethernet  HWaddr 00:CC:CC:ED:F3:B0
      inet addr:10.1.2.7  Bcast:10.1.2.255  Mask:255.255.255.0
      inet6 addr: fe80::20c:29ff:feed:f3b0/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:63036 errors:0 dropped:0 overruns:0 frame:0
      TX packets:12508 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:6047120 (5.7 MiB)  TX bytes:1439770 (1.3 MiB)
      Interrupt:59 Base address:0x2000

lo        Link encap:Local Loopback
      inet addr:127.0.0.1  Mask:255.0.0.0
      inet6 addr: ::1/128 Scope:Host
      UP LOOPBACK RUNNING  MTU:16436  Metric:1
      RX packets:28378 errors:0 dropped:0 overruns:0 frame:0
      TX packets:28378 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:2410342 (2.2 MiB)  TX bytes:2410342 (2.2 MiB)

sit0      Link encap:IPv6-in-IPv4
      NOARP  MTU:1480  Metric:1
      RX packets:0 errors:0 dropped:0 overruns:0 frame:0
      TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

I'm still suspicious of something, somehow, ARP-related, but note that an arp -n does not even list 10.1.2.3 at all--no ARP cache entry, even after the telnet, ping, etc., of 10.1.2.3. It's just as if Host A suddenly "thinks" it is 10.1.2.3 when nothing in its configuration says so.

And, what's more...so does Host B (same exact behavior), Host C, Host D, and so on. Each one of those hosts, when testing, turns out to be connecting back to itself when directed to connect to 10.1.2.3.

Finally decided to reboot one of them just to see if there was a running kernel/ARP issue (read: pretty much desperation). Not solved, though for a few seconds or maybe a minute after reboot the host doesn't "think" it's this other IP, but then it resumes said issue.

So I have a plethora of hosts, each suddenly "thinking" it is 10.1.2.3 but can find no configuration, no ARP table, nor anything else indicating they've assumed a binding to that IP. Meanwhile, it appears that a bunch of Windows hosts on the LAN are not having this problem--they're not exhibiting this behavior (may or may not be relevant).

Even tried moving one host to a different switch in case there was a switch issue. Not solved.

In case it's relevant, another common attribute is that all these CentOS hosts are actually VMs running under VMWare Server--but not all on the same host hardware. Multiple hypervisor hosts in play.

Dan
  • 61
  • 2
  • Have you tried `tcpdump` or similar to capture a packet trace - especially the resolution/connecting phase? Given what you've got listed, about the only thing I can think of off the top of my head is some sort of fruity NAT going on. – Sobrique Apr 28 '14 at 19:57
  • `ifconfig` is deprecated. Please give us the output of `ip addr` instead. And in your case, the output of `ip route get 10.1.2.3` might be important. Consider also giving the output of `ip route list table all`. – BatchyX Apr 28 '14 at 20:18
  • What's the output of `ip neigh`? – sciurus Apr 28 '14 at 20:24
  • Sure enough tcpdump yields what ip addr notes: `[root@hostA tmp]# ip addr 1: lo: mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet 10.1.2.2/32 scope global lo inet 10.1.2.3/32 scope global lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever` Didn't know about ip addr--am too rusty. Sure enough, there it is--it thinks it's 10.1.2.3. No idea how. – Dan Apr 28 '14 at 20:36
  • And `[root@vmdev01 tmp]# ip route get 192.168.10.53 local 192.168.10.53 dev lo src 192.168.10.53 cache mtu 16436 advmss 16396 hoplimit 64` – Dan Apr 28 '14 at 20:39
  • Does HostA have a load balancer installed or something? – hookenz Apr 28 '14 at 20:39
  • HostA doesn't have a load balancer, no, but one is "in play" inasmuch as we do have one on the network brokering access to other hosts--and in fact 10.1.2.3 is the pool IP address for that (which is how this mess got started). – Dan Apr 28 '14 at 20:44
  • To make matters more confusing, sorry--above in my output that shows 192.168.10.53, that's actually the IP in question. I had tried to sanitize a little for my examples but 10.1.2.3 is really 192.168.10.53, too late to edit that comment though. – Dan Apr 28 '14 at 20:46
  • ARRG. Welp, found it. Turns out I was wrong above--this host (and several others) was cloned from hosts that DO sit behind the load balancer and "10.1.2.3" (actually 192.168.10.53) is in fact assigned right here in /etc/rc.local: ip addr add dev lo 192.168.10.53/32 That shouldn't have been the case on this host and now I see why, it being a clone. Now the real problem we were troubleshooting still exists--some other host is also grabbing that IP. But the scope of my question on this thread is answered. Sorry, thanks very much for the help. – Dan Apr 28 '14 at 20:52

0 Answers0