2

More specifically, at the core of erl, what algorithm is used to understand presence and availability of other nodes? How does it handle network partitioning? Are all the nodes just constantly pinging each other?

For example, if there are two nodes, and the network cable is pulled, how does it decide what to do? Presumably one node should go idle as it's orphaned, while the other carries on, otherwise you get a split-brain behavior..

In reading up on paxos and raft, it seems like it must be doing leader election internally, but I can't seem to find any comprehensible explanation -- I left my PhD in my other pants.. Can anyone explain this voodoo in english?

XeroxDucati
  • 5,130
  • 2
  • 37
  • 66
  • possible duplicate of [How is the detection of terminated nodes in Erlang working? How is net\_ticktime influencing the control of node liveness in Erlang?](http://stackoverflow.com/questions/24061270/how-is-the-detection-of-terminated-nodes-in-erlang-working-how-is-net-ticktime) – Nathaniel Waisbrot Jul 15 '14 at 14:16

1 Answers1

2

I've got the same question here and it was answered with this very good article where you can find detailed explanation to your question. The main idea is: Erlang nodes in distributed mode are connected through mesh network. They monitor each other through pings that are done in period of time decided by net_tick_time constant. Pings are used to detect network splits or halted nodes unable to communicate. Other failures such as VM crashes, cable unplugs are detected immediately (within few ms) through underlying network connection.

Community
  • 1
  • 1
Zuzana
  • 132
  • 6
  • While this may answer the question, it is better to provide the actual information here and not just a link. [Link-only answers are not considered good answers and will probably be deleted](http://stackoverflow.com/help/deleted-answers). – elixenide Jul 15 '14 at 14:56
  • Sorry, I did not know about this, I will give some brief explanation then, thanks. – Zuzana Jul 15 '14 at 15:03
  • Interesting, thanks! This certainly explains how they wire up, but it doesn't quite explain what happens in the background if things go unavailable. For instance, if process 1 is on node 1, and process 2 is on node 2, if node 2 goes down, node 1 would pick up both process 1 and 2 (as I understand it). But, if its a network split, wouldnt i wind up with p1 and p2 running on n1 and also on n2, since both see the other one die? – XeroxDucati Jul 15 '14 at 19:25
  • I think you might be mixing two concepts here. The detection of node/process failures happens via a combination of listening for OS error messages (TCP connection closed), listening for messages sent by erlang when a node goes down, and as a last resort measure, timeouts / net_tick_time. – Alexander Jul 16 '14 at 07:23
  • The "process moving around" thing sounds like OTP's concept of a "distributed application". Note that it's entire applications, and not just processes and that they don't really get moved, but just started at another node (loss of data). And yes, it's very possible to get two or more instances of an application running. That's why there is a "takeover" functionality that allows you to deal with scenarios where e.g. n1 and n2 are reconnected and realize they're both running the application. – Alexander Jul 16 '14 at 07:24
  • @XeroxDucati Processes do not magically move around when a node fails. If a node fails, then every process on it is seen to send an EXIT/DOWN message to all processes on other nodes linked or monitoring those on the node that died. The supervision tree is likely written to recreate those elsewhere, but that's up to the developer. At a minimum, the state that the processes had on the node that went down *that iteration* of their execution is lost. The purpose of the supervision tree isn't to overcome CAP, it is to automatically return to a known, stable state and carry on. – zxq9 Oct 14 '14 at 22:41