Payara - Hazelcast cluster node picks the wrong network interface

Question

When starting Payara cluster, one of the nodes binds to the wrong IP address (internal IP address of the docker which is installed locally on the node).

What is the proper way of letting know the Payara Cluster instance node which address it should bind to?

Node 1 log:

[2017-12-04T11:35:06.512+0800] [Payara 4.1] [INFO] [] [com.hazelcast.internal.cluster.impl.MulticastJoiner] [tid: _ThreadID=16 _ThreadName=RunLevelControllerThread-1512358500010] [timeMillis: 1512358506512] [levelValue: 800] [[
[172.17.0.1]:5900 [dev] [3.8.5] 
Members [1] {
    Member [172.17.0.1]:5900 - 9be6669e-b853-44c0-9656-8488d3e1031b this
}
]]

Node 2 log:

[2017-12-04T11:35:06.771+0800] [Payara 4.1] [INFO] [] [com.hazelcast.internal.cluster.impl.MulticastJoiner] [tid: _ThreadID=17 _ThreadName=RunLevelControllerThread-1512358500129] [timeMillis: 1512358506771 [levelValue: 800] [[
[10.4.0.86]:5900 [dev] [3.8.5] 
Members [1] {
    Member [10.4.0.86]:5900 - e3f9dd48-58b9-45f9-88fc-6b0feaedd78f this
}
]]

I have tested the cluster itself and it works properly on machines with the only one interface (without docker installed).

I have found issues that are related to my case, but was not able to adapt them in Payara Cluster setup:

• Hazelcast cluster over AWS using Docker
• Configuring a two node hazelcast cluster - avoiding multicast

Meaning, suggestion to use the local property: -Dhazelcast.local.localAddress=[yourCorrectIpGoesHere] - works, but in case of cluster environment with centralized management of the nodes config - I do not see how to set the different JVM properties for each of the nodes.

Submitting custom hazelcast-config.xml via the "Override configuration file" could be an option, but it means that full configuration should be done via this file, what makes it not super handy to manage, but currently looks like this is the only option that potentially can help here.

Thanks!

Answer 1

Payara server doesn't expose this configuration option directly. Using the system property hazelcast.local.localAddress is the preferred option. However, you shouldn't set it as a JVM option like you did with

-Dhazelcast.local.localAddress=...

Instead, add the system property using the server page in the Admin Console. On the Properties tab go to System properties tab and add a new property with the variable name hazelcast.local.localAddress and override value set to the IP address of the interface you want Hazelcast to bind to.

This way, the configuration is applied during runtime without any server restart needed and should also be propagated to other instances in the cluster if you also set the property for cluster instances. For those, instead of going to the server page you would go to the configuration of each instance and set the system property there.