We are using Haproxy on top of Mesos cluster, We are doing dynamic reloads for Haproxy based on marathon events (50-100 times in a day). We have nearly 300 applications that are running on Mesos (300 virtual hosts in Haproxy). When we do dynamic reloads, Haproxy is taking long time to reload Haproxy, we observed that for 50 applications takes 30-40secs to reload Haproxy. We have a single config file for Haproxy, when we do reload all the applications are getting reloaded (Front-ends), this causing downtime of all applications. Is there anyway to reduce the downtime and impact on end-users.
We tried this scenario,
"http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html"
By this if user requests while reload, the requests are queued and serving after reload.
But if we do multiple reloads one after another, HaProxy's old processes persist even after reloading the HaProxy service, this is causing serious issues.
root 7816 0.1 0.0 20024 3028 ? Ss 03:52 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 6778
root 7817 0.0 0.0 20024 3148 ? Ss 03:52 0:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 6778
Is there any solution stop the previous process once after it serving the request.
Can we separate the configurations based on front-ends like in Nginx, so that only those apps will effect if there is any changes in backend.
Thank you.
