Server comunication fail on same IP block

Question

Good morning, I own 7 servers in the same DC.

Of these, 4 of them are unable to communicate with each other: each of these 4 machines does not communicate with the other 3 via SSH and there is no response to ping ("Destination Host Unreachable").

These 4 servers are all identical, purchased on the same day. Communication between these 4 machines and the other 3 machines that I own works regularly (ping, ssh, etc.).

Of course, there are no firewall blocks or anything else because, as I said, communication with other machines works and I tested on just-installed machines.

IMPORTANT: all 4 servers belong to the SAME IP block xx.xx.xx.* (example 123.123.123.56, 123.123.123.59, 123.123.123.68, etc.)

Some suggestion, please?

root@myserver [~] # ip link show  
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1  
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00  
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000  
    link/ether 00:22:4d:7a:b7:c0 brd ff:ff:ff:ff:ff:ff  

root@myserver [~] # ip addr show  
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00  
    inet 127.0.0.1/8 scope host lo  
       valid_lft forever preferred_lft forever  
    inet6 ::1/128 scope host   
       valid_lft forever preferred_lft forever  
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000  
    link/ether 00:22:4d:7a:b7:c0 brd ff:ff:ff:ff:ff:ff  
    inet 838.309.278.48/24 brd 838.309.278.255 scope global eth0  
       valid_lft forever preferred_lft forever  
    inet6 2001:41d0:8:7830::1/128 scope global   
       valid_lft forever preferred_lft forever  
    inet6 fe80::222:4dff:fe7a:b7c0/64 scope link   
       valid_lft forever preferred_lft forever  

root@myserver [~] # ip route show  
default via 838.309.278.254 dev eth0   
838.309.278.0/24 dev eth0 proto kernel scope link src 838.309.278.48

iptables-save -c

root@myserver [~] # iptables-save -c
# Generated by iptables-save v1.4.21 on Tue Nov 21 08:53:14 2017
*mangle
:PREROUTING ACCEPT [697331:228627666]
:INPUT ACCEPT [697331:228627666]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [654668:238129552]
:POSTROUTING ACCEPT [627123:232560477]
COMMIT
# Completed on Tue Nov 21 08:53:14 2017
# Generated by iptables-save v1.4.21 on Tue Nov 21 08:53:14 2017
*nat
:PREROUTING ACCEPT [77156:3042292]
:INPUT ACCEPT [72516:2727270]
:OUTPUT ACCEPT [41719:6614113]
:POSTROUTING ACCEPT [14174:1045038]
COMMIT
# Completed on Tue Nov 21 08:53:14 2017
# Generated by iptables-save v1.4.21 on Tue Nov 21 08:53:14 2017
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
:ALLOWIN - [0:0]
:ALLOWOUT - [0:0]
:DENYIN - [0:0]
:DENYOUT - [0:0]
:INVALID - [0:0]
:INVDROP - [0:0]
:LOCALINPUT - [0:0]
:LOCALOUTPUT - [0:0]
:LOGDROPIN - [0:0]
:LOGDROPOUT - [0:0]
[0:0] -A INPUT -s 8.8.4.4/32 ! -i lo -p tcp -m tcp --dport 53 -j ACCEPT
[0:0] -A INPUT -s 8.8.4.4/32 ! -i lo -p udp -m udp --dport 53 -j ACCEPT
[0:0] -A INPUT -s 8.8.4.4/32 ! -i lo -p tcp -m tcp --sport 53 -j ACCEPT
[954:68688] -A INPUT -s 8.8.4.4/32 ! -i lo -p udp -m udp --sport 53 -j ACCEPT
[0:0] -A INPUT -s 8.8.8.8/32 ! -i lo -p tcp -m tcp --dport 53 -j ACCEPT
[0:0] -A INPUT -s 8.8.8.8/32 ! -i lo -p udp -m udp --dport 53 -j ACCEPT
[0:0] -A INPUT -s 8.8.8.8/32 ! -i lo -p tcp -m tcp --sport 53 -j ACCEPT
[7982:866312] -A INPUT -s 8.8.8.8/32 ! -i lo -p udp -m udp --sport 53 -j ACCEPT
[660785:221345203] -A INPUT ! -i lo -j LOCALINPUT
[27610:6347463] -A INPUT -i lo -j ACCEPT
[122886:73580584] -A INPUT ! -i lo -p tcp -j INVALID
[119005:73544720] -A INPUT ! -i lo -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
[0:0] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 20 -j ACCEPT
[25:1216] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 21 -j ACCEPT
[210:9016] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 22 -j ACCEPT
[4098:231644] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 25 -j ACCEPT
[7:344] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 53 -j ACCEPT
[216:11384] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 80 -j ACCEPT
[17:820] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 110 -j ACCEPT
[17:848] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 143 -j ACCEPT
[49:2176] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 443 -j ACCEPT
[10:464] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 465 -j ACCEPT
[21:1128] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 587 -j ACCEPT
[9:400] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 993 -j ACCEPT
[4:180] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 995 -j ACCEPT
[42:2172] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2030 -j ACCEPT
[0:0] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2031 -j ACCEPT
[0:0] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2082 -j ACCEPT
[0:0] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2083 -j ACCEPT
[4:220] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2086 -j ACCEPT
[11:660] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2087 -j ACCEPT
[0:0] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2095 -j ACCEPT
[0:0] -A INPUT ! -i lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2096 -j ACCEPT
[0:0] -A INPUT ! -i lo -p udp -m conntrack --ctstate NEW -m udp --dport 20 -j ACCEPT
[0:0] -A INPUT ! -i lo -p udp -m conntrack --ctstate NEW -m udp --dport 21 -j ACCEPT
[11:717] -A INPUT ! -i lo -p udp -m conntrack --ctstate NEW -m udp --dport 53 -j ACCEPT
[61724:2149570] -A INPUT ! -i lo -p icmp -m icmp --icmp-type 8 -m limit --limit 1/sec -j ACCEPT
[2:96] -A INPUT ! -i lo -p icmp -m icmp --icmp-type 0 -m limit --limit 1/sec -j ACCEPT
[5:320] -A INPUT ! -i lo -p icmp -m icmp --icmp-type 11 -j ACCEPT
[24:2583] -A INPUT ! -i lo -p icmp -m icmp --icmp-type 3 -j ACCEPT
[4627:313990] -A INPUT ! -i lo -j LOGDROPIN
[0:0] -A OUTPUT -d 8.8.4.4/32 ! -o lo -p tcp -m tcp --dport 53 -j ACCEPT
[954:68688] -A OUTPUT -d 8.8.4.4/32 ! -o lo -p udp -m udp --dport 53 -j ACCEPT
[0:0] -A OUTPUT -d 8.8.4.4/32 ! -o lo -p tcp -m tcp --sport 53 -j ACCEPT
[0:0] -A OUTPUT -d 8.8.4.4/32 ! -o lo -p udp -m udp --sport 53 -j ACCEPT
[0:0] -A OUTPUT -d 8.8.8.8/32 ! -o lo -p tcp -m tcp --dport 53 -j ACCEPT
[7982:581323] -A OUTPUT -d 8.8.8.8/32 ! -o lo -p udp -m udp --dport 53 -j ACCEPT
[0:0] -A OUTPUT -d 8.8.8.8/32 ! -o lo -p tcp -m tcp --sport 53 -j ACCEPT
[0:0] -A OUTPUT -d 8.8.8.8/32 ! -o lo -p udp -m udp --sport 53 -j ACCEPT
[618122:231132078] -A OUTPUT ! -o lo -j LOCALOUTPUT
[0:0] -A OUTPUT ! -o lo -p tcp -m tcp --dport 53 -j ACCEPT
[9:673] -A OUTPUT ! -o lo -p udp -m udp --dport 53 -j ACCEPT
[20:2011] -A OUTPUT ! -o lo -p tcp -m tcp --sport 53 -j ACCEPT
[11:2510] -A OUTPUT ! -o lo -p udp -m udp --sport 53 -j ACCEPT
[27610:6347463] -A OUTPUT -o lo -j ACCEPT
[132012:58252558] -A OUTPUT ! -o lo -p tcp -j INVALID
[193611:60395750] -A OUTPUT ! -o lo -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 20 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 21 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 22 -j ACCEPT
[4:240] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 25 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 53 -j ACCEPT
[130:7800] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 80 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 110 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 113 -j ACCEPT
[27:1620] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 443 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2030 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2031 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2082 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2083 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2086 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2087 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2095 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 2096 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 587 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 993 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p tcp -m conntrack --ctstate NEW -m tcp --dport 995 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p udp -m conntrack --ctstate NEW -m udp --dport 20 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p udp -m conntrack --ctstate NEW -m udp --dport 21 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p udp -m conntrack --ctstate NEW -m udp --dport 53 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p udp -m conntrack --ctstate NEW -m udp --dport 113 -j ACCEPT
[5292:402192] -A OUTPUT ! -o lo -p udp -m conntrack --ctstate NEW -m udp --dport 123 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p icmp -m icmp --icmp-type 0 -j ACCEPT
[38:3192] -A OUTPUT ! -o lo -p icmp -m icmp --icmp-type 8 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p icmp -m icmp --icmp-type 11 -j ACCEPT
[0:0] -A OUTPUT ! -o lo -p icmp -m icmp --icmp-type 3 -j ACCEPT
[27545:5569075] -A OUTPUT ! -o lo -j LOGDROPOUT
[260880:81590254] -A ALLOWIN -s 93.65.24.152/32 ! -i lo -j ACCEPT
[222056:70086439] -A ALLOWOUT -d 93.65.24.152/32 ! -o lo -j ACCEPT
[0:0] -A DENYIN -s 201.255.85.155/32 ! -i lo -j DROP
[0:0] -A DENYOUT -d 201.255.85.155/32 ! -o lo -j LOGDROPOUT
[348:15236] -A INVALID -m conntrack --ctstate INVALID -j INVDROP
[0:0] -A INVALID -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j INVDROP
[0:0] -A INVALID -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,PSH,ACK,URG -j INVDROP
[0:0] -A INVALID -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -j INVDROP
[0:0] -A INVALID -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -j INVDROP
[0:0] -A INVALID -p tcp -m tcp --tcp-flags FIN,RST FIN,RST -j INVDROP
[0:0] -A INVALID -p tcp -m tcp --tcp-flags FIN,ACK FIN -j INVDROP
[0:0] -A INVALID -p tcp -m tcp --tcp-flags PSH,ACK PSH -j INVDROP
[0:0] -A INVALID -p tcp -m tcp --tcp-flags ACK,URG URG -j INVDROP
[13:1032] -A INVALID -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j INVDROP
[361:16268] -A INVDROP -j DROP
[660785:221345203] -A LOCALINPUT ! -i lo -j ALLOWIN
[190499:76290936] -A LOCALINPUT ! -i lo -j DENYIN
[618122:231132078] -A LOCALOUTPUT ! -o lo -j ALLOWOUT
[226687:66385063] -A LOCALOUTPUT ! -o lo -j DENYOUT
[1676:68728] -A LOGDROPIN -p tcp -m tcp --dport 23 -j DROP
[0:0] -A LOGDROPIN -p udp -m udp --dport 23 -j DROP
[0:0] -A LOGDROPIN -p tcp -m tcp --dport 67 -j DROP
[0:0] -A LOGDROPIN -p udp -m udp --dport 67 -j DROP
[0:0] -A LOGDROPIN -p tcp -m tcp --dport 68 -j DROP
[90:31050] -A LOGDROPIN -p udp -m udp --dport 68 -j DROP
[20:960] -A LOGDROPIN -p tcp -m tcp --dport 111 -j DROP
[4:272] -A LOGDROPIN -p udp -m udp --dport 111 -j DROP
[0:0] -A LOGDROPIN -p tcp -m tcp --dport 113 -j DROP
[0:0] -A LOGDROPIN -p udp -m udp --dport 113 -j DROP
[5:224] -A LOGDROPIN -p tcp -m tcp --dport 135:139 -j DROP
[10:780] -A LOGDROPIN -p udp -m udp --dport 135:139 -j DROP
[395:20388] -A LOGDROPIN -p tcp -m tcp --dport 445 -j DROP
[0:0] -A LOGDROPIN -p udp -m udp --dport 445 -j DROP
[0:0] -A LOGDROPIN -p tcp -m tcp --dport 500 -j DROP
[8:3424] -A LOGDROPIN -p udp -m udp --dport 500 -j DROP
[1:40] -A LOGDROPIN -p tcp -m tcp --dport 513 -j DROP
[0:0] -A LOGDROPIN -p udp -m udp --dport 513 -j DROP
[1:40] -A LOGDROPIN -p tcp -m tcp --dport 520 -j DROP
[2:104] -A LOGDROPIN -p udp -m udp --dport 520 -j DROP
[2022:84660] -A LOGDROPIN -p tcp -m limit --limit 30/min -j LOG --log-prefix "Firewall: *TCP_IN Blocked* "
[376:101298] -A LOGDROPIN -p udp -m limit --limit 30/min -j LOG --log-prefix "Firewall: *UDP_IN Blocked* "
[0:0] -A LOGDROPIN -p icmp -m limit --limit 30/min -j LOG --log-prefix "Firewall: *ICMP_IN Blocked* "
[2415:187980] -A LOGDROPIN -j DROP
[2:120] -A LOGDROPOUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 30/min -j LOG --log-prefix "Firewall: *TCP_OUT Blocked* " --log-uid
[18149:3725921] -A LOGDROPOUT -p udp -m limit --limit 30/min -j LOG --log-prefix "Firewall: *UDP_OUT Blocked* " --log-uid
[0:0] -A LOGDROPOUT -p icmp -m limit --limit 30/min -j LOG --log-prefix "Firewall: *ICMP_OUT Blocked* " --log-uid
[27545:5569075] -A LOGDROPOUT -j REJECT --reject-with icmp-port-unreachable
COMMIT
# Completed on Tue Nov 21 08:53:14 2017

ip neigh show

root@myserver [~] # ip neigh show
fe80::da24:bdff:fe90:740 dev eth0 lladdr d8:24:bd:90:07:40 router STALE
2001:41d0:8:78ff:ff:ff:ff:ff dev eth0 lladdr 00:05:73:a0:00:00 router STALE
838.309.278.51 dev eth0  FAILED
838.309.278.64 dev eth0  FAILED
838.309.278.57 dev eth0  FAILED
838.309.278.250 dev eth0 lladdr 00:07:b4:00:00:01 STALE
838.309.278.254 dev eth0 lladdr 00:07:b4:00:00:01 REACHABLE

arptables & ebtables

root@myserver [~] # arptables-save
*filter
:INPUT ACCEPT
:OUTPUT ACCEPT
:FORWARD ACCEPT

root@myserver [~] # ebtables-save
# Generated by ebtables-save v1.0 on Tue Nov 21 10:59:44 CET 2017

Adding the result of these commands (for at least one if all are similar) in the question would certainly help: `ip link show` , `ip addr show` , `ip route show` , `iptables-save -c` — A.B, Nov 21 '17 at 07:50
And whatever you add to your question, do not redact it. For this kind of question, the devil tends to be in the details, so we need to see them. — MadHatter, Nov 21 '17 at 07:52
Good morning here the results of the suggested commands. I can also report result of "iptables-save -c" if needed, I skipped because quite long: — Carlo Queirolo, Nov 21 '17 at 07:59
The `state UP` result from the output of `ip link show` show that at least a cable is plugged in and the switch port isn't shut. That's good. — HBruijn, Nov 21 '17 at 08:13
One last "silly" question. Do you have any rules visible beside the 3 builtins with the commands `arptables-save` or `ebtables-save` ? I really think it's PVLAN but just to be sure... Oh and are .51 .64 .57 the 3 other servers ? Also what's the role of .250 (which appears to be working) ? — A.B, Nov 21 '17 at 09:56
Sorry, I don't know "what's the role of .250". Yes, the listed IP are of the other three servers! I updated the question with arptables and ebtables — Carlo Queirolo, Nov 21 '17 at 10:02

A.B · Accepted Answer · 2017-11-21T10:24:33.683

According to the result of ip neigh show you can't reach the three other servers (IPs ending in .51 .57 .64) at the link layer level. This really looks like port isolation aka PVLAN.

Your Data Center doesn't want some of its client/customers managing to access some of its other clients/customers' servers just because they are in the same LAN, for liability reasons. So it's using this isolation method for simplicity. The result is the same as with "Client Isolation" on Wifi.

What you should do is contact the Data Center, prove you're the same owner for the 4 servers, and ask them some exception to the isolation rules so they can talk together. In the wikipedia link from above, that would be to have them put in the same "Community" secondary VLAN instead of simply "Isolated".

Of course there's no way to completely prove this is what happens beside asking them, but everything fits.

Server comunication fail on same IP block

1 Answers1