strategy for distributing erlang otp nodes behind haproxy

Question

I have a standard erlang application build upon otp principles. I plan to place my erlang nodes in production as explained below:

1. receive all traffic on a public ip (haproxy)
2. haproxy them to one of the available backend erlang node

All this works very well however in case of session based http transactions, chances are one of the erlang node receives a request, who's session related processes resides on another erlang node.

My questions are related to best strategy for handling such cases:

1. first option is to configure haproxy to balance based upon source (i.e. ip address), this will always guarantee all requests for a session from a single ip goes to same backend erlang node
2. second option is to configure haproxy to balance based on some session related cookie parameter (which basically gives me same as case 1))
3. finally, since my erlang nodes can very well communicate with each other, it is also possible to simply configure haproxy to balance in roundrobin fashion, and when an erlang node receives a request, whose session processes reside on another erlang node, it can internally communicate with the erlang node using rpc:call() and serve the request.

1),2) are pretty much straighforward to use and setup, however I read and tested that source/cookie/url_param based balancing doesn't ensure equal balancing among backends.

3) is achievable using mnesia replication power inside erlang. With 3) I will probably end up with a case where one erlang node communicates internally to another erlang node before it finally serves response to a request.

I would like to know what is preferable and why? Will 3) be a better choice in long run considering my otp application handles a lot of real-time data (xmpp protocol).

Answer 1

I think this largely depends on how much load one of your nodes can handle in terms of sessions. The IP and cookie hashing, at least in terms of HTTP, does a pretty good job of keeping balance with a large number of short lived sessions.

So for example, there might be many customers behind a single IP (cookie based hashing should help this). If a single node can only handle a relatively small amount of simultaneous sessions then a node might get overloaded. Also if sessions are long lived the imbalance of hashing might be pronounced. If on the other hand, the number of sessions is large and short lived, the relative significance of one node having a few more connections likely doesn't really matter that much.

In theory I would also expect that with some form of source or cookie based balancing you will get better cache hit ratios. One other thing to consider is that if you have say 3 nodes, then if one node goes down each of the other nodes would have ~ 16% more load on each of them even with perfect load balancing. So you should factor provisioning in as well.

If you can handle a lot of requests on a single node, then third option to me sounds like you are solving for a problem you don't have and are not likely to have when a far more simple option is feasible.