4

I have a pair of servers setup as high availability load balancers/reverse proxies. Each runs Ubuntu 12.04 x64 Server, Varnish, Heartbeat and Pacemaker, with Varnish load balancing traffic to back-end servers.

If either of the load balancers falls over, Heartbeat/Pacemaker transfers a group of virtual IPs over to the other server, and traffic flow resumes. This bit works fine.

What I haven't accounted for is if Varnish isn't running on either server. It's currently possible to stop Varnish without triggering any sort of action from Heartbeat/Pacemaker. I'd like the absense of an operational Varnish on the current server to trigger a move to the backup (rather than attempting to restart Varnish), but I'm struggling to find any sort of guidance online. Can anyone help?

Edit post-Daff's assistance:

I ended up with something a little different from my original request: Pacemaker attempts to restart Varnish once, and if that fails, it moves all resources to the passive node.

My setup is two servers, serverA (active) and serverB (passive). I'll assume that the messaging layer (Heartbeat or Corosync) is already setup and working. To allow Pacemaker to control Varnish, we need to fix Ubuntu's Varnish init script:

sudo vim /etc/init.d/varnish

Replace:

--start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- \

in the start_varnish_d() function with:

--start --quiet --pidfile ${PIDFILE} --oknodo --exec ${DAEMON} -- \

so it works in accordance with the rules outlined here. Now setup a basic Pacemaker cluster on serverA with a two virtual IPs:

sudo crm configure property no-quorum-policy=ignore
sudo crm configure property stonith-enabled=false
sudo crm configure primitive virtual_ip_1 ocf:heartbeat:IPaddr params ip="192.168.1.134" nic="eth1" cidr_netmask="24" broadcast="192.168.1.255" op monitor interval="10s" timeout="20s"
sudo crm configure primitive virtual_ip_2 ocf:heartbeat:IPaddr params ip="192.168.1.135" nic="eth1" cidr_netmask="24" broadcast="192.168.1.255" op monitor interval="10s" timeout="20s"

Add a primitive for Varnish, providing a monitoring frequency, and generous timings for starting and stopping Varnish:

sudo crm configure primitive varnish lsb:varnish op monitor interval="10s" timeout="20s" op
start interval="0" timeout="15s" op stop interval="0" timeout="15s"

Group the varnish primitive with the virtual IPs, so Pacemaker migrates all resources to the passive node in the event of a failure:

sudo crm configure group cluster virtual_ip_1 virtual_ip_2 varnish

Set migration-threshold, so Pacemaker will tolerate two failures before moving all resources to the passive node. For Varnish, this means one initial failure, plus one failed restart attempt:

sudo crm_attribute --type rsc_defaults --attr-name migration-threshold --attr-value 2

Set a failure timeout. This seems to do two things:

  1. Gives Pacemaker the time for one Varnish restart attempt before migrating to the passive node.

  2. Prevents the failed node being marked as failed after 30s, allowing resources to be moved back to it without manually having to run crm resource cleanup varnish after a failure. This is good thing for my setup, as I don't have any weightings set on the nodes, but it could be a really bad idea in a different environment.

sudo crm_attribute --type rsc_defaults --attr-name failure-timeout --attr-value 30s

And that's it, but see Daff's answer below for comments about stickiness, which I didn't end up using. The only downside I can see is that if you manually put a node into standby, Pacemaker will shutdown Varnish on that node, thus clearing the in-memory cache. For me, that isn't a particularly big deal.

jetboy
  • 912
  • 2
  • 11
  • 25
  • Your terminology is confusing... you don't use *both* heartbeat and pacemaker to monitor and failover services, you use one or the other. – womble Jul 24 '12 at 23:18
  • 1
    @womble To be fair, the whole Linux HA terminology is confusing because of its history. Heartbeat 1.0 became Heartbeat 2.0 where it supported more than a simple two-node HA setup, and in Heartbeat 2.x the resource manager part was spun off into Pacemaker, which supported not only Heartbeat (now just a messaging layer again) but other cluster engines as well. Corosync, formally OpenAIS, is one of them. Wikipedia contains a bit of history on both. But it is possible and supported to use Pacemaker with Heartbeat instead of Corosync. Although I don't know why anyone would do that today. – daff Jul 24 '12 at 23:38
  • Yes, Pacemaker with Heartbeat as a messaging layer. Why Heartbeat and not Corosync? Familiarity with Heartbeat, and at the time this was originally setup, a comparative lack of documentation on Corosync. – jetboy Jul 25 '12 at 00:25

1 Answers1

3

Your cluster architecture confuses me, as it seems you are running services that should be cluster-managed (like Varnish) standalone on two nodes at the same time and let the cluster resource manager (CRM) just juggle IP addresses around.

What is it you want to achieve with your cluster setup? Fault tolerance? Load balancing? Both? Mind you, I am talking about the cluster resources (Varnish, IP addresses, etc), not the backend servers to which Varnish distributes the load.

To me it sounds like you want an active-passive two-node cluster, which provides fault tolerance. One node is active and runs Varnish, the virtual IP addresses and possibly other resources, and the other node is passive and does nothing until the cluster resource manager moves resources over to the passive node, at which point it becomes active. This is a tried-and-true architecture that is as old as time itself. But for it to work you need to give the CRM full control over the resources. I recommend following Clusters from Scratch and modelling your cluster after that.

Edit after your updated question: your CIB looks good, and once you patched the Varnish init script so that repeated calls to "start" return 0 you should be able to add the following primitive (adjust the timeouts and intervals to your liking):

primitive p_varnish lsb:varnish \
    op monitor interval="10s" timeout="15s" \
    op start interval="0" timeout="10s" \
    op stop interval="0" timeout="10s"

Don't forget to add it to the balancer group (the last element in the list):

group balancer eth0_gateway eth1_iceman_slider eth1_iceman_slider_ts \
    eth1_iceman_slider_pm eth1_iceman_slider_jy eth1_iceman eth1_slider \
    eth1_viper eth1_jester p_varnish

Edit 2: To decrease the migration threshold add a resource defaults section at the end of your CIB and set the migration-threshold property to a low number. Setting it to 1 means the resource will be migrated after a single failure. It is also a good idea to set resource stickiness so that a resource that has been migrated because of node failure (reboot or shutdown) does not automatically get migrated back later when the node is available again.

rsc_defaults $id="rsc-options" \
    resource-stickiness="100" \
    migration-threshold="1"
daff
  • 4,809
  • 2
  • 28
  • 27
  • You've described the architecture I'm already running (better than I did). The key point is that the cluster isn't currently managing Varnish, and I'd like to know how to a. Monitor Varnish, b. Use a Varnish failure to trigger moving the IPs to the passive node. – jetboy Jul 25 '12 at 00:16
  • @jetboy can you post the output of "crm configure show" so that I can see what you are dealing with? Essentially you should add a new "primitive" that represents Varnish (probably using lsb:varnish as resource type) and use the "op monitor" directive so that the CRM regularly checks if Varnish is still running. – daff Jul 25 '12 at 00:34
  • Amended question with crm configure show output. OK, I've read up on LSB at http://linux-ha.org/wiki/LSB_Resource_Agents and it looks like Ubuntu's Varnish init script isn't compliant, as it returns result: 1 when attempting to start an already running instance. – jetboy Jul 25 '12 at 01:01
  • Don't use LSB init scripts as RAs. They're pretty much guaranteed to be insufficiently robust to be useful in Pacemaker. If an RA is flaky, you can pretty much guarantee far less reliability than if you just ran a single machine. – womble Jul 25 '12 at 09:32
  • Oh, the hours I've spent bringing init scripts up to LSB specs. One would think that in 2012 the package maintainers would have finally understood that an init script has to do more than just start and stop something. The LSB specs have been around for many years now. – daff Jul 25 '12 at 12:59
  • @jetboy The Varnish init script is mostly ok, you should just patch it so that calling "start" repeatedly exits in 0 and not 1. The actions "stop" and "status" work well enough as far as I can see (not using Varnish myself) so you should be good. – daff Jul 25 '12 at 13:00
  • OK, setup the varnish primitive, added it the to IP group, patched the init script, and it's being monitored. Stopping varnish on the active node causes Pacemaker to try to restart Varnish. If it can't restart it, it just carries on trying. So, thanks to your help it works a lot better than it did, but I'm still short on the key requirement from my original question: "I'd like the absense of an operational Varnish on the current server to trigger a move to the backup (rather than attempting to restart Varnish)" i.e. Broken Varnish on the active node should trigger failover to the passive node. – jetboy Jul 25 '12 at 17:04
  • @jetboy In a correctly configured cluster failure to start a resource means the resource will be failed over to another cluster node. So if your Varnish is broken on the active node and cannot be (re-)started (init script returns anything other than 0 on "start") it, and its group, will be moved to the passive node. It seems you observe different behaviour? Try setting `migration-threshold` to a lower number in the resource defaults section and see if that makes a difference. I'll update my answer above. – daff Jul 25 '12 at 18:17
  • Aye, I've butchered the VCL to create a fail on Varnish start, and it's exiting with a 1 correctly, but Pacemaker just goes into a restart loop. I'll try your additions and post back in a bit... – jetboy Jul 25 '12 at 19:35
  • Nice :) I've ended up with a bit of a hybrid approach, allowing Pacemaker one chance at restarting Varnish before migrating nodes. I've left the stickiness as is, as it's working how I want it to. Daff, thanks for all your help and patience. – jetboy Jul 25 '12 at 21:23
  • Updated original post with a how-to based on Daff's answer. – jetboy Jul 25 '12 at 22:29
  • @jetboy Great to hear! But I think you need to switch "exit 0" and "exit 1" in your howto above, no? A better way to patch the init script is to add the `--oknodo` option to the `start-stop-daemon` command line. Also, the init script is rather bad because it exits successfully even when the daemon binary is not found or not executable: `test -x $DAEMON || exit 0`. Should exit with 1 instead. – daff Jul 25 '12 at 23:08
  • Cheers Daff. Amend made (and tested). I'm going to quit while I'm ahead, but out of interest, is your last point still an issue even with this amend? – jetboy Jul 25 '12 at 23:49
  • Yes, it is, because if the binary `/usr/sbin/varnishd` were renamed, removed or not executable the init script would not complain. It would simply exit with 0, indicating to the CRM that the resource was started when in fact it was not! This underlines womble's earlier point that LSB init scripts are generally badly written and should only be used with care as resource agents in Pacemaker. – daff Jul 26 '12 at 00:15
  • After running with this a while, I've got to echo womble's comments. The Varnish LSB/Pacemaker combination doesn't cope with restarts or reboots at all well (even after hacking), and gets locked in a non-working state. As a consequence, I've decided to take Varnish out of the HA group. I also finally got around to changing the messaging layer from Heartbeat to CoroSync, which is relatively pain-free. – jetboy Aug 17 '12 at 17:30