We use Keepalived (running on RHEL 7) to manage shared IP addresses on our 3 HAProxy servers. Each server has 2 interfaces, one with a public IP and one with a private IP. We are migrating from a pair of Kemp LoadMaster LM-3000 appliances.
We noticed that two of the three HAProxy systems are logging many lines every second for bogus VRRP packet received on em2 !!!.
This is 1 second of logs. I cut out the times and process numbers to save space.
haproxy01 Keepalived_vrrp: VRRP_Instance(haproxy::fqdn) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
haproxy01 Keepalived_vrrp: bogus VRRP packet received on em2 !!!
haproxy01 Keepalived_vrrp: VRRP_Instance(haproxy::fqdn) ignoring received advertisment...
haproxy00 Keepalived_vrrp: VRRP_Instance(haproxy::device) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
haproxy00 Keepalived_vrrp: bogus VRRP packet received on em2 !!!
haproxy00 Keepalived_vrrp: VRRP_Instance(haproxy::device) ignoring received advertisment...
haproxy00 Keepalived_vrrp: VRRP_Instance(haproxy::support) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
haproxy00 Keepalived_vrrp: bogus VRRP packet received on em2 !!!
haproxy00 Keepalived_vrrp: VRRP_Instance(haproxy::support) ignoring received advertisment...
haproxy00 Keepalived_vrrp: VRRP_Instance(haproxy::whiffle) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
haproxy00 Keepalived_vrrp: bogus VRRP packet received on em2 !!!
haproxy00 Keepalived_vrrp: VRRP_Instance(haproxy::whiffle) ignoring received advertisment...
haproxy00 Keepalived_vrrp: VRRP_Instance(haproxy::www) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
haproxy00 Keepalived_vrrp: bogus VRRP packet received on em2 !!!
haproxy00 Keepalived_vrrp: VRRP_Instance(haproxy::www) ignoring received advertisment...
haproxy00 Keepalived_vrrp: VRRP_Instance(haproxy::wwwdev) IPSEC-AH : invalid IPSEC HMAC-MD5 value. Due to fields mutation or bad password !
haproxy00 Keepalived_vrrp: bogus VRRP packet received on em2 !!!
haproxy00 Keepalived_vrrp: VRRP_Instance(haproxy::wwwdev) ignoring received advertisment...
haproxy02 doesn't log any odd traffic.
haproxy00.example.com:em1 -> 172.24.0.200,em2 -> 192.0.2.32haproxy01.example.com:em2 -> 172.24.0.201,em1 -> 192.0.2.29haproxy02.example.com:em1 -> 172.24.0.202,em2 -> 192.0.2.24kemp00.example.com:em1 -> 172.24.0.48,em2 -> 192.0.2.59kemp01.example.com:em1 -> 172.24.0.49,em2 -> 192.0.2.60- "Floating"
kemp.example.com:172.24.0.50 - "Floating"
kemp-public.example.com:192.0.2.63
Note that haproxy01 has em1 and em2 reversed compared to the other two.
The haproxy* systems are set to use unicast VRRP instead of multicast (sample of the configuration from haproxy00 the others are exactly the same except for the changed interface name for haproxy01 and a different priority. haproxy02 is the MASTER):
vrrp_instance haproxy::fqdn {
interface em1
state BACKUP
virtual_router_id 199
priority 100
advert_int 1
garp_master_delay 5
authentication {
auth_type AH
auth_pass csvrp199
}
virtual_ipaddress {
172.24.0.199/24 dev em1
}
virtual_routes {
metric 5 to default via 172.24.0.1
}
unicast_src_ip 172.24.0.200
unicast_peer {
172.24.0.201
172.24.0.202
}
}
vrrp_instance haproxy::device {
interface em2
state BACKUP
virtual_router_id 15
priority 100
advert_int 1
garp_master_delay 5
authentication {
auth_type AH
auth_pass csvrrp15
}
virtual_ipaddress {
192.0.2.7/26 dev em2
}
virtual_routes {
metric 5 to default via 192.0.2.1
}
unicast_src_ip 192.0.2.32
unicast_peer {
192.0.2.24
192.0.2.29
}
}
vrrp_instance haproxy-csweb:haproxy::support { ... }
vrrp_instance haproxy-csweb:haproxy::whiffle { ... }
vrrp_instance haproxy-csweb:haproxy::www { ... }
vrrp_instance haproxy-csweb:haproxy::wwwdev { ... }
We know VRRP is working between the haproxy systems because we can shutdown haproxy02 and traffic moves to the next highest priority system.
tcpdump shows the traffic we think is causing this problem. The addrs advertised from one LM to the other appear to be obfuscated in some way, since none of them are our real addresses.
[root@haproxy00 ~]# tcpdump -vvvvvni em2 vrrp
tcpdump: listening on em2, link-type EN10MB (Ethernet), capture size 262144 bytes
13:54:09.672726 IP (tos 0x10, ttl 255, id 51772, offset 0, flags [DF], proto VRRP (112), length 56)
192.0.2.59 > 224.0.0.18: vrrp 192.0.2.59 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 1, authtype none, intvl 1s, length 36, addrs(7): 127.73.197.82,124.192.126.111,231.25.226.215,113.220.143.181,197.101.63.203,152.246.226.65,46.55.62.80
[root@haproxy01 ~]# tcpdump -vvvvvni em2 vrrp
tcpdump: listening on em2, link-type EN10MB (Ethernet), capture size 262144 bytes
13:54:36.739262 IP (tos 0x10, ttl 255, id 50547, offset 0, flags [DF], proto VRRP (112), length 56)
172.24.0.48 > 224.0.0.18: vrrp 172.24.0.48 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 1, authtype none, intvl 1s, length 36, addrs(7): 120.18.52.8,8.96.198.173,44.237.204.205,99.139.163.15,10.76.116.67,163.0.175.114,121.54.183.104
So errors are logged on em2 of each server for each vrrp_instance, but not for the instances on em1, and also not at all on haproxy02.
We are trying to stop these errors that are being logged because they hide more important errors and make tailing the log files impossible.
Separately, we also have iptables which I would think should be blocking the multicast traffic, but doesn't seem to be:
[root@haproxy00 ~]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 /* 000 accept all icmp */
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* 001 accept all to lo interface */
REJECT all -- 0.0.0.0/0 127.0.0.0/8 /* 002 reject local traffic not on loopback interface */ reject-with icmp-port-unreachable
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* 003 accept related established rules */ state RELATED,ESTABLISHED
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 multiport dports 80,443
ACCEPT tcp -- 172.16.0.0/12 0.0.0.0/0 multiport dports 22 /* 203 allow internal sshd:22 */ state NEW
ACCEPT 112 -- 192.0.2.24 0.0.0.0/0 /* 226 Allow vrrp from 192.0.2.24 */
ACCEPT 112 -- 192.0.2.29 0.0.0.0/0 /* 226 Allow vrrp from 192.0.2.29 */
ACCEPT 112 -- 172.24.0.201 0.0.0.0/0 /* 226 Allow vrrp from 172.24.0.201 */
ACCEPT 112 -- 172.24.0.202 0.0.0.0/0 /* 226 Allow vrrp from 172.24.0.202 */
DROP all -- 0.0.0.0/0 0.0.0.0/0 /* 999 drop all */
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
