HAProxy/Keepalived vrrp set up in AWS EC2 with EIP (Basic EC2)

Question

I am a newbie to CentOS/Linux/HAProxy/Keepalived, so I have had a lot to learn over the past few days!

I have 2 CentOS 6.5 servers in Amazon AWS running on their own EC2 Micro instances. I also have 1 Elastic IP set up. Each instance has the following installed:

• Amazon EC2 API Tools
• HAProxy
• Keepalived
• Apache / httpd
• Java OpenJDK 1.6

I have got HAProxy installed and configured on both instances. For testing purposes I installed Apache to serve a basic index.html page so I could test that HAProxy was carrying out the LB task and round robin is working fine.

What I am trying to achieve, is using Keepalived to monitor the haproxy process on LB1 (Has EIP assigned), if it fails or stops for some reason, it calls a script which uses the AWS EC2 API Tools to reassign an Elastic IP to the second Loadbalancer to take over..

I have googled and looked at the documentation to create a script that achieves this, however, when I test it and the HAProxy service is stopped, it does not run the script..

LB1 - Contents of keepalived.conf

vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 2
weight 2
}

vrrp_instance VI_1 {
interface eth0
state MASTER
virtual_router_id 51
priority 101
vrrp_unicast_bind 10.xx.xx.01 # The IP of the host this file is stored on
vrrp_unicast_peer 10.xx.xx.23 # The IP of the other host
advert_int 1
track_script {
chk_haproxy
}
notify_fault /etc/keepalived/vrrp.sh
}

Output from Tailing the log

Jun 18 12:03:11 ip-10-xx-xx-xx Keepalived_vrrp[25199]: Opening file '/etc/keepalived/keepalived.conf'.
Jun 18 12:03:11 ip-10-xx-xx-xx Keepalived_vrrp[25199]: Configuration is using : 60618 Bytes
Jun 18 12:03:11 ip-10-xx-xx-xx Keepalived_vrrp[25199]: Using LinkWatch kernel netlink reflector...
Jun 18 12:03:11 ip-10-xx-xx-xx Keepalived_vrrp[25199]: VRRP sockpool: [ifindex(2), proto(112), fd(10,11)]
Jun 18 12:03:11 ip-10-xx-xx-xx Keepalived_vrrp[25199]: VRRP_Script(chk_haproxy) succeeded
Jun 18 12:03:12 ip-10-xx-xx-xx Keepalived_vrrp[25199]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jun 18 12:03:13 ip-10-xx-xx-xx Keepalived_vrrp[25199]: VRRP_Instance(VI_1) Entering MASTER STATE
Jun 18 12:03:19 ip-10-xx-xx-xx Keepalived_vrrp[25199]: VRRP_Script(chk_haproxy) failed

My first impression is that it has detected the haproxy service failure, but it doesnt seem to execute the vrrp.sh script. (contents below)

#vrrp.sh
#!/bin/bash
cd $EC2_BASE/tools/bin
#DisAssociate EIP from this instance.
./ec2-disassociate-address  54.xx.xx.xx
#Mapping EIP to secondary server
./ec2-associate-address 54.xx.xx.xx -i [instance-id]

If I run the vrrp.sh script by itself on the server, it runs the commands and successfully re-assigns the EIP. Just not when it is called from the keepalived.conf script.

Any ideas!? I've been scratching my head all day and the laptop is about to be launched out the window..

Answer 1

The notify_fault script is invoked when keepalived enters the fault state.

You want to configure your script using

notify_master /etc/keepalived/vrrp.sh

Answer 2

Can you please paste your backup.conf and slave.conf files?

Also, you should check these two articles:

http://www.trk7.com/blog/keepalived-instance-not-entering-failed-state/ or http://comments.gmane.org/gmane.linux.keepalived.devel/4102 ?

Answer 3

First to determine if the script is or is not executing, I would add the following line directly below #!/bin/bash.

echo "----------vrrp.sh running ------" >> /var/log/messages 

or something whatever you want... Just to log the fact that the application has seen the "state" transition from backup to master or whatever, and has in fact executed the script as intended.

You could put it at the end of the script as well, the point is to just make sure the the script was called.

Test it again, and then cat /var/log/messages, and look for your echo... you should see it, if you don't then it could be any number of things, permissions, user account that is running, ensure the script executable, and so on... likely based on your initial comment I suspect it is executing its just not doing what you expect.

So if you do see your echo, as I expect you will, then I would look at your environment configuration, what you could do, just for testing, is export your JAVA_HOME along with your EC2 variables in the script... add them after the echo line you just added. Obviously this is just for testing purposes and once you confirm its all working as expected, then you should remove this from the script, and configure your environment appropriately. My final suggestion here is to change your command... You can disassociate and associate the EIP in a single command.

So now your script will look something like this:

#!/bin/bash
#vrrp.sh

echo "----------vrrp.sh running ------" >> /var/log/messages

cd /usr/local/ec2/ec2-api-tools-1.7.5.1/bin/

export JAVA_HOME="/usr/lib/jvm/jre-1.8.0-openjdk-1.8.0.65-0.b17.el6_7.x86_64/"

export EC2_BASE="/usr/local/ec2/ec2-api-tools-1.7.5.1/"
export EC2_URL=https://ec2.us-east-1.amazonaws.com
export AWS_ACCESS_KEY=AKIA3ROU5M3FQIQH4BNL
export AWS_SECRET_KEY=InBiqtOCfaJhCb3u3jE6gpmkW5shgjP8N++/7Huv

#Those are obviously not my access/secret keys, just providing an example

./ec2-associate-address -a eipalloc-s234523 -i i-023ksdfj --allow-reassociation

Final note:

Be sure that the IAM User account you are authenticating with has the necessary IAM user account policy attached... an example of a policy is below you will have to go to policies to create it, and then once created just got to the users and the specific IAM user account you are authenticating with and attach the policy you just created unless you are using an FULL ACCESS account in which case, I would recommend against that. and to setup an IAM user account with the policy outlined below:

arn:aws:iam::107890181863:policy/MOVE-EIP or whatever.

{
"Version": "2015-1-1",
"Statement": [
    {
        "Action": [
            "ec2:AssociateAddress",
            "ec2:DescribeAddresses",
            "ec2:AllocateAddress",
            "ec2:DisassociateAddress"
        ],
        "Effect": "Allow",
        "Resource": "*"
    }
  ]
}