HTTP Ports are Filtered on New Virtual IPs in LVS (Linux Virtual Server) Load Balancer

Question

I inherited a Linux Virutal Server (LVS) load balancer on CentOS 5.10. Its been running with no issues for quite some time with no worries.

Now when I add a new virutal IP (VIP), all HTTP traffic is "filtered" to that port.

For example: Here's nmap output for an existing VIP:

nmap 10.150.200.141
Starting Nmap 5.51.6 ( http://nmap.org ) at 2014-10-13 14:55 EDT
Nmap scan report for 10.150.200.141
Host is up (0.014s latency).
Not shown: 995 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
111/tcp  open  rpcbind
443/tcp  open  https
1556/tcp open  veritas_pbx
Nmap done: 1 IP address (1 host up) scanned in 0.28 seconds

And here's the nmap output for a VIP I just added:

nmap -Pn 10.150.200.47
Starting Nmap 5.51.6 ( http://nmap.org ) at 2014-10-13 14:58 EDT
Nmap scan report for 10.150.200.47
Host is up (0.011s latency).
Not shown: 995 closed ports
PORT     STATE    SERVICE
22/tcp   open     ssh
80/tcp   filtered http
111/tcp  open     rpcbind
443/tcp  filtered https
1556/tcp open     veritas_pbx
Nmap done: 1 IP address (1 host up) scanned in 1.31 seconds

I should note that the config for the new VIP is a copy of the original VIP; I just changed the name, eth0, and IP.

Another thing to note, the real servers in the new VIP were previously in the original VIP and worked fine there. Now I just need to break them out on a VIP of their own for testing.

I tried another brand new VIP with a different real server (unrelated to the ones previously mentioned) and got the same result.

Updating with iptables output:

/sbin/iptables -L --line -n -v
Chain INPUT (policy ACCEPT 106M packets, 16G bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 107M packets, 16G bytes)
num   pkts bytes target     prot opt in     out     source               destination

Updating with ipvsadm output:

ipvsadm -L -n -t 10.150.200.47:80
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.150.200.47:80 wlc
  -> 10.150.200.247:80            Route   50     0          0

Updating with iptables output from the real servers:

Chain RH-Firewall-1-INPUT (2 references)
num  target     prot opt source               destination
1    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
2    ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0           icmp type 255
3    ACCEPT     esp  --  0.0.0.0/0            0.0.0.0/0
4    ACCEPT     ah   --  0.0.0.0/0            0.0.0.0/0
5    ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0           udp dpts:161:162
6    ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
7    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80
8    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:443
9    ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpts:161:162
10   ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
11   REJECT     all  --  0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Any thoughts or suggestions for debugging would be appreciated.

Answer 1

Check the firewall configuration:

/sbin/iptables -L --line -n -v

Chances are you will need to allow traffic to that VIP. For example:

/sbin/iptables -I INPUT <line#> -d 10.150.200.47 -p tcp -m tcp --dport 80 -j ACCEPT 
/sbin/iptables -I INPUT <line#> -d 10.150.200.47 -p tcp -m tcp --dport 443 -j ACCEPT 

Also, verify that the real servers are accepting connections, for example:

$ sudo ipvsadm -L -n -t 10.150.200.47:80
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.150.200.47:80  wrr
  -> 192.168.100.11:80            Route   1      56         145       
  -> 192.168.100.12:80            Route   1      49         159       

Edit: Adding things to check on the real server(s):

Assuming you are using direct routing the virtual IP will need to be configured on the loopback interface of the real server(s). When the connection is forwarded from the load balancer to the secondary server it will respond back out on eth0 as normal. What this configuration would look like:

ifconfig lo:0 10.150.200.47 netmask 255.255.255.255

Or as a network script:

$ cat /etc/sysconfig/network-scripts/ifcfg-lo:0
DEVICE=lo:0
IPADDR=10.150.200.47
NETMASK=255.255.255.255
ONBOOT=yes

Additionally you will need to make kernel parameter changes via sysctl:

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

Additionally you'll need to change how ARP is announced and responds to requests (more information):

net.ipv4.conf.eth0.arp_ignore = 1
net.ipv4.conf.eth0.arp_announce = 2
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

Also, configure rp filtering:

net.ipv4.conf.eth0.rp_filter = 1 # Works for CentOS 5
or
net.ipv4.conf.eth0.rp_filter = 2 # Works for CentOS 6+

The default rp_filter setting can vary between kernel versions, so make sure you choose the correct setting. More information.