ICMP echos and replies arrive, but no ping (kvm hosts and guests)

Question

This are the steps I took building my setup:

1. I have 2 computers (both x86 running Ubuntu 14.04 LTS).
2. I install 2 Virtual Machines (VMs) - one on each machine.
3. Both VMs are Ubuntu Server 14.04 LTS.
4. On one x86 I run one VM using kvm. On the other x86 I run the other VM using VirtualBox.
5. I connect both x86s through a Router.

====================================================

(ubuntu1)[KVM](X86,1)------{Router}------(X86,2)[VirtualBox](ubuntu2)

====================================================

Now, I allow IPv4 forwarding on all 4 machines (change on /etc/sysctl.conf, and using sysctl -p), and set all 4 machines to have a single NIC (eth0), and set it in /etc/network/interfaces to manual).

Next, I start by assigning addresses to the NICs of the two x86s , set default gateways and ping around. Everything works great. After that I assign an address to one VM (instead of the x86 on which this VM is running on). Again - works great. And finally - set an IP address to the second VM (again, instead of the x86 on which the second VM is running on).

At a certain point - ping doesn't work. When I check the packets, I can see that one VM answers and the other does not. I checked this in the following ways:

1. I choose one VM (let's say ubuntu1) and I ping and tcpdump on this machine (ping [IP_ADDRESS_ubuntu2] & followed by tcpdump -i eth0)
2. Connect the wire between the x86s to a hub (duplicating the packets) and wireshark-ing it.

The result is always the same - I can see one VM registers echos of ICMP packets to the other (this is the one that replies) and the other VM registers echos and replies from the other VM.

So - ICMP packets actually arrive the machine, but ping shows nothing!

Here's an example of the last setup:

====================================================

(55.55.55.55)[KVM]-----------{Router}-----------[VirtualBox](22.22.22.22)

====================================================

Running ping and tcpdump on 55.55.55.55:

root@55.55.55.55:/home# ping 22.22.22.22 &
root@55.55.55.55:/home# PING 22.22.22.22 (22.22.22.22) 56(84) bytes of data.
root@55.55.55.54:/home# tcpdump -i eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
07:10:11.758872 IP 55.55.55.55 > 22.22.22.22: ICMP echo request, id 5499, seq 6, length 64
07:10:12.758785 IP 55.55.55.55 > 22.22.22.22: ICMP echo request, id 5499, seq 7, length 64
07:10:13.766994 IP 55.55.55.55 > 22.22.22.22: ICMP echo request, id 5499, seq 8, length 64
07:10:14.767637 IP 55.55.55.55 > 22.22.22.22: ICMP echo request, id 5499, seq 9, length 64

Running ping and tcpdump on 22.22.22.22:

root@22.22.22.22:/home# ping 55.55.55.55 &
root@22.22.22.22:/home# PING 55.55.55.55 (55.55.55.55) 56(84) bytes of data.
root@22.22.22.22:/home# tcpdump -i eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:58:01.883915 IP 22.22.22.22 > 55.55.55.55: ICMP echo request, id 4099, seq 8, length 64
15:58:01.884390 IP 55.55.55.55 > 22.22.22.22: ICMP echo reply, id 4099, seq 8, length 64
15:58:02.885795 IP 22.22.22.22 > 55.55.55.55: ICMP echo request, id 4099, seq 9, length 64
15:58:02.886298 IP 55.55.55.55 > 22.22.22.22: ICMP echo reply, id 4099, seq 9, length 64

Note, though, that the machine "knows" where the other machine is:

root@55.55.55.55:/home/rad# ip route get 22.22.22.22
22.22.22.22 via 55.55.55.1 dev eth2  src 55.55.55.55
    cache

Ideas?

UPDATE 1

output of various commands:

root@55.55.55.55:/home# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

root@55.55.55.55:/home# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT

root@55.55.55.55:/home# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         55.55.55.1      0.0.0.0         UG        0 0          0 eth0
55.55.55.0      0.0.0.0         255.255.255.0   U         0 0          0 eth0

UPDATE 2

A change of an IP address on one side (55.55.55.55 --> 55.55.55.54) gets an ICMP reply - meaning ping works.

A change of an IP address on the other-side makes no difference.

root@55.55.55.55:/home/rad# ifconfig eth0 55.55.55.54/24
root@55.55.55.55:/home/rad# ping 22.22.22.22
PING 22.22.22.22 (22.22.22.22) 56(84) bytes of data.
64 bytes from 22.22.22.22: icmp_seq=1 ttl=63 time=1.05 ms
64 bytes from 22.22.22.22: icmp_seq=2 ttl=63 time=0.568 ms
64 bytes from 22.22.22.22: icmp_seq=3 ttl=63 time=0.692 ms
64 bytes from 22.22.22.22: icmp_seq=4 ttl=63 time=0.597 ms

switching back to the 55.55.55.55 IP address, denies the ICMP reply again.

Directions

Several directions I'm thinking in, but still haven't found an answer:

• the HW address of a neighbor machine is listed somewhere and associated with an IP address, preventing the packets from being routed to the correct location
• the arp cache values Tresh1, Tresh2 and Tresh3 are not being set correctly, preventing the garbage collection of iproute2 from cleaning the registries - thus enabling the arp cache to be replaced
• some kernel feature related to routing dumps the packets, confusing between the host and the guest machines