2

I have an OTP application and I'd like to distribute it over several nodes. I'm quite new to Erlang so what I'm going to ask might sound really basic, but I couldn't find anything online.

To start with, say I want to have two nodes running at the same time. One is the primary one and one is the backup. If the primary falls over, the backup takes over. When that happens, the backup becomes the primary and the node that was the primary, once the application is restarted, becomes the backup. The important thing is that the application has a state and it must not be lost.

My idea is that the primary receives the messages from the user and forwards them to the backup, so they both run at the same time. Only the primary responds to the user though. If the primary node falls over, the backup can take over since it should have the same state.

Hopefully the user will not notice anything.

What's the best way to achieve this?

Thank you very much.

Edit Should have clarified that I'd like the system to work in case of a net split, so persisting the data on hard-drive doesn't look like a viable solution.

satoshi
  • 3,963
  • 6
  • 46
  • 57

1 Answers1

2

In the erlang terms this is known as failover and takeover. Erlang makes it as easy as specifying the nodes in the kernel configuration file

Here is a sample from the official documentation

[{kernel,
 [{distributed, [{myapp, 5000, [cp1@cave, {cp2@cave, cp3@cave}]}]},
 {sync_nodes_mandatory, [cp2@cave, cp3@cave]},
  {sync_nodes_timeout, 5000}
   ]
  }
  ].

Here my app is primary. When it fails it is restarted at cp2 or cp3. 5000 here is the latency time that must elapse before application is restarted. It is optional and can be zero. As far as preserving state is concerned the only thing I can think of is using a database like mnesia to store your states in. The distributed applications are also explained in learn you some erlang

Edit based on comment

You can store the state of your application in mnesia in a disc_copy manner. This means that the all the data will be in ram but a copy will be saved on disc for backup purposes. Mnesia can store erlang terms directly. There is no serialization. If you are using gen_* like behavior you can easily store the state and then restart it from the same state on a different node. It sounds daunting at first but I think after a few trials you should be able to get this done.

Regarding the change in configuration files... yes it seems the only way to add new nodes is to edit configuration files. But I guess this is not as painful as you might imagine. Remember that you can access erlang at run time. Maybe create a module that saves the state and call it from the command line at runtime. Then edit the configuration files and restart the application.

Also note that the configuration file is different for each node. So if you know in advance how many nodes you are running on you may not need to do any of the steps mentioned above.

Akshat Jiwan Sharma
  • 15,430
  • 13
  • 50
  • 60
  • 1
    Thank you @AkshatJiwanSharma. I've been looking at what you said before and I know I can do it that way but I don't know how to maintain the state. Probably storing it on disk is the only solution then. Another concern I have is that I have to specify the host in the kernel configuration file (@cave in your case). What if I want to give this application to run it on another computer, will I have to manually change the kernel configuration file to reflect the host change? Thanks! – satoshi Mar 16 '14 at 12:00
  • No problem @satoshi. My comment was getting too big so I edited the answer. I want to add that erlang gives you all the tools to design a distributed application but it you are new, like myself, it may take a while to get a hang of things. Try to run multiple nodes on a single machine before experimenting on different computers. – Akshat Jiwan Sharma Mar 16 '14 at 12:34
  • Thank you Thank you @AkshatJiwanSharma. The problem with this solution is that it wouldn't work in case of a net split, persisting the state on hard-drive only on the primary node. I'm looking for a solution where the state is update in real time on the backup node, so that it's ready to startup as soon as the primary node goes down. Thanks – satoshi Mar 16 '14 at 14:33
  • I am not too sure but I don't think that it is possible to both maintain state and guarantee availability. By the way you can manually replicate states on different nodes. Check [this answer](http://stackoverflow.com/questions/9423832/access-distributed-mnesia-database-from-different-nodes). – Akshat Jiwan Sharma Mar 16 '14 at 14:48