0

I use a ReplicatedMap<String, String> in my Hazelcast cluster. Each node has its key (hazelcastInstance.getLocalEndpoint().getUuid()) and some data (JSON string) in this map. Using this map I implemented a service such that each node has its data store, can access data of other nodes, and observe changes.

However I struggle to get the split brain behaviour right.

I would like to receive an entryAdded or entryUpdated event on my EntryListener attached to the ReplicatedMap when a split brain merge happens. However, this is not the case. I assume this is by design (?).

What solutions could you imagine to solve this?

Ideas:

  • MembershipListener can detect the split brain merge. Problem: The data in ReplicatedMap is not necessarily ready yet - it could be ready before or after the memberAdded event is called.

  • Implement MergePolicy, e.g.:

    public Object merge(String mapName, ReplicatedMapEntryView mergingEntry, ReplicatedMapEntryView existingEntry) {
    // every node knows its own values! (somehow existingEntry and mergingEntry are swapped?)
    // mergingEntry seems to be the destination (current local value)
    // existingEntry seems to be the value from the other member
    if (mergingEntry.getKey().equals(hazelcastUuid)) {
        return mergingEntry.getValue();
    } else {
        return existingEntry.getValue();
    }
}

    Problem: This works fine when split brain merging two nodes. However, using four nodes (two nodes per machine on two different machines), it annihilates values and seems to behave randomly. I suspect that I understood something generally wrong because this approach is not working as hoped. Because of this, I didn't analyze the behavior of the four nodes yet - also it's quite time consuming to analyze four log files.

a1337q
  • 770
  • 3
  • 10
  • 20
  • The statement - "Using this map I implemented a service such that each node has its data store, can access data of other nodes, and observe changes" is confusing because if you are using Replicated Map then the node (which I believe is referring to Hazelcast member in this case) does not perform a remote operation; instead, it returns the data from its local memory. Also, each node in the cluster has all data. Therefore there is no question of a node accessing data of other nodes. – wildnez May 19 '17 at 14:20
  • Membership listeners may or may not detect split brain, for a split brain happens when the cluster is partitioned on network. A member can leave the cluster for various other reasons, hence membership listener is not a good choice. Can you elaborate on what you are trying to achieve by trying to attach listener to a replicated map? What are you expecting to see during split-brain and merge? – wildnez May 19 '17 at 14:21

0 Answers0