2

I am using HAProxy socket communication to put my backend into maintenance mode before bringing the web server down for maintenance via this command:

echo "disable server cluster-01/app-01-4" | sudo socat stdio /tmp/haproxy.sock

The problem is that when I mark the backend available again via the socket enable command, HAProxy does not wait for any successful health checks before sending requests to the backend. So we end up losing some of the requests before HAProxy realizes the server is still down, and takes it back off line.

Is there a way to tell HAProxy to wait until the backend is healthy prior to enabling it? Or do I have to make my startup scripts smart enough to monitor the server itself prior to telling HaProxy to enable the backend again? This would obviously be less than ideal.

I'm currently doing this:
echo "enable server cluster-01/app-01-4" | sudo socat stdio /tmp/haproxy.sock

Is there something akin to this that I can use instead? echo "enable server cluster-01/app-01-4 WAIT_FOR_SUCCESSFUL_HEALTH_CHECK" | sudo socat stdio /tmp/haproxy.sock

Edit: I am using HAProxy version 1.5x

WickedElephant
  • 123
  • 1
  • 4

3 Answers3

1

Using set server proxy-name/server-name health down prior to putting the server back in service with enable server ... appears to do what you need.

The server would eventually be taken out of service by subsequent health checks, but that doesn't help if the server was healthy when you took it out, but not healthy when you bring it back in, since HAProxy appears to assume that the server will be in the same health state when enabled that it was in when it was disabled.

Michael - sqlbot
  • 22,658
  • 2
  • 63
  • 86
0

You can move to haproxy 1.6.3 and use the new agent-check functionality. This also lets you drain hosts out as you take them out of rotation.

Zypher
  • 37,405
  • 5
  • 53
  • 95
  • Thank you very much for your help Zypher. It does seem like 1.6.3 provides the functionality I seek out of the box. `The word "up" sets back the server's operating state as UP if health checks also report that the service is accessible.` In the end, I am going with Michael - sqlbot's answer because it works with HAProxy 1.5 as a workaround. – WickedElephant Mar 04 '16 at 18:53
0

At Loadbalancer.org we do this is a slightly complicated way :-).

Before we restart HAProxy we check all of the current states and check for any servers that are DOWN or MAINT.

Then we BLOCK SYN packets (handles the dropped traffic on reload issue) Then we restart HAProxy

Then we manually set all the states to DOWN or MAINT (based on what they were 0.01 seconds ago..)

Then we re-enable SYN packets...

Giving us the required result of seamless restarts with no downtime or lost packets.

If you can't be bothered with ALL the above then just set everything to DOWN state after a reload ..do it fast and cross your fingers :-).

v1.6 has some new state-full restart code but I haven't researched it yet.

Malcolm turnbull
  • 827
  • 6
  • 6
  • Thanks for your reply, but it sounds like you are talking about restarting HAProxy, which is not what my question is about. I am talking about doing rolling restarts of my application/web servers, and how to keep HAProxy from sending requests to them until they are healthy once I re-enable them in HAProxy. – WickedElephant Mar 04 '16 at 18:40
  • Sorry , I misunderstood however it should still a similar process: Like the guy said previously put the server in MAINT state. Do the maintenance and then set the server to READY/DOWN state... and as soon as you bring the server back online and it passes a health check it will change to UP state. Or use the feedback agent in file monitor mode: http://www.loadbalancer.org/blog/open-source-windows-service-for-reporting-server-load-back-to-haproxy-load-balancer-feedback-agent – Malcolm turnbull Mar 06 '16 at 17:00