Hazelcast AtomicLong Data loss When multiple member left

Question

Hazelcast fails when multiple members disconnect from the cluster. My scenario is so basic and my configuration has 3 bakcup option(it does not work). I have 4 members in a cluster and i use AtomicLong API to save my key->value. When all members are alive, everything is perfect. However, some data loss occurs when i kill 2 members at the same time(without waiting for a while). My member counts are always 4. is there any way to avoid this kind of data loss?

  Config config = new Config();
  NetworkConfig network = config.getNetworkConfig();
  network.setPort(DistributedCacheData.getInstance().getPort());
  config.getCacheConfig("default").setBackupCount(3);
  JoinConfig join = network.getJoin();
  join.getMulticastConfig().setEnabled(false);
  join.getTcpIpConfig().setEnabled(true);  
  config.setNetworkConfig(network);
  config.setInstanceName("member-name-here");

Thanks.

ali · Accepted Answer · 2019-12-12T10:27:35.467

3

IAtomicLong has hard-coded 1 sync backup, you cannot configure it to have more than 1 backup. What you are doing is configuring Cache with 3 backups.

Below is a example demonstrates multiple member disconnect for IMap

Config config = new Config();
config.getMapConfig("myMap").setBackupCount(3);
HazelcastInstance[] instances = {
        Hazelcast.newHazelcastInstance(config),
        Hazelcast.newHazelcastInstance(config),
        Hazelcast.newHazelcastInstance(config),
        Hazelcast.newHazelcastInstance(config)
};
IMap<Integer, Integer> myMap = instances[0].getMap("myMap");
for (int i = 0; i < 1000; i++) {
    myMap.set(i, i);
}
System.out.println(myMap.size());
instances[1].getLifecycleService().terminate();
instances[2].getLifecycleService().terminate();
System.out.println(myMap.size());

edited Dec 12 '19 at 10:27

answered Dec 11 '19 at 15:55

ali

876
4
9

When i check HazelCast official Doc, I saw backup section under MAP API and i just wantted to try it for IAtomicLong. As you said, i did not work. How can i avaid data loss when multiple member disconnect from cluster? there should be a way at minimum. – Okay Atalay Dec 12 '19 at 06:05
As I've said there is no way for you to configure more than 1 backup for `IAtomitlong` data-structure but you can configure other data-structures such as `IMap` up to 6 backups. In that case `IMap` will not loose data if multiple members disconnect from the cluster. I've updated my answer to demonstrate this. – ali Dec 12 '19 at 10:24
I used IAtomicLong because my data structure was so basic and IAtomicLog work more efficient/performance. i have to convert all structure to IMap again and make some tunings. – Okay Atalay Dec 12 '19 at 11:40
btw, I have tried your example and when main is completed. HZ Threads still continue to work and throws exception in infinite loop. Is there any config to limit attempt. (Reason: SocketException[Connection refused: no further information to address) – Okay Atalay Dec 12 '19 at 11:48

Hazelcast AtomicLong Data loss When multiple member left

1 Answers1