Hazelcast network interfaces

Question

I have got 3 interfaces like below:

enp10s0: inet 192.168.1.101  netmask 255.255.255.0  broadcast 192.168.1.255
lo: inet 127.0.0.1  netmask 255.0.0.0 
tun0: inet 192.168.216.122  netmask 255.255.255.255 ...

and hazelcast 3.6.4 configuration simplified to code below

@Configuration
public class HazelcastConfig {

private static final Logger LOGGER = LoggerFactory.getLogger(HazelcastConfig.class);

private Environment environment;

@Autowired
public void setEnvironment(Environment environment) {
    this.environment = environment;
}

/**
 * @return hazelcast server side configuration
 * @see "http://docs.hazelcast.org/docs/3.6/manual/html-single/index.html"
 */
@Bean(name = "hazelcastNetworkConfig")
public Config getHazelcastConfig() {
    final Config config = new Config();
    config.getSecurityConfig().setEnabled(true);
    config.setProperty("hazelcast.initial.min.cluster.size", "1");
    config.setProperty("hazelcast.socket.bind.any", "false");
    config.setProperty("hazelcast.socket.server.bind.any", "false");
    config.setProperty("hazelcast.socket.client.bind.any", "false");
    config.setProperty("hazelcast.logging.type", "slf4j");
    final GroupConfig groupConfig = new GroupConfig();
    final String hazelcastGroupName = environment.getRequiredProperty("hazelcast.name");
    LOGGER.info("Configuring hazelcast with group name: " + hazelcastGroupName);
    groupConfig.setName(hazelcastGroupName);
    groupConfig.setPassword(environment.getRequiredProperty("hazelcast.password"));
    config.setGroupConfig(groupConfig);
    final NetworkConfig networkConfig = new NetworkConfig();
    networkConfig.setPortAutoIncrement(true);
    networkConfig.setReuseAddress(true);
    final JoinConfig joinConfig = networkConfig.getJoin();
    joinConfig.getAwsConfig().setEnabled(false);
    joinConfig.getMulticastConfig().setEnabled(false);
    joinConfig.getTcpIpConfig().setEnabled(true);
    joinConfig.getTcpIpConfig().setConnectionTimeoutSeconds(5);
    final String[] members = "host.domain.eu:8080".split(",");
    for (String member : members) {
        joinConfig.getTcpIpConfig().addMember(member);
        LOGGER.info("Configuring hazelcast with tcp ip member: " + member);
    }
    final String hazelcastInterfaces = "192.168.1.*";
    if (hazelcastInterfaces == null || hazelcastInterfaces.isEmpty()) {
        LOGGER.info("Configuring hazelcast without specified interfaces");
    } else {
        final InterfacesConfig interfacesConfig = new InterfacesConfig();
        interfacesConfig.setEnabled(true);
        LOGGER.info("Configuring hazelcast with specified interfaces");
        final String[] interfaces = hazelcastInterfaces.split(",");
        for (String hazelcastInterface : interfaces) {
            LOGGER.info("Configuring hazelcast with interface: " + hazelcastInterface);
            interfacesConfig.addInterface(hazelcastInterface);
        }
        networkConfig.setInterfaces(interfacesConfig);
    }
    config.setNetworkConfig(networkConfig);
    final String hazelcastInstanceName = environment.getRequiredProperty("hazelcast.instanceName");
    LOGGER.info("Configuring hazelcast with instance name: " + hazelcastInstanceName);
    config.setInstanceName(hazelcastInstanceName);
    return config;
}

@Bean
public HazelcastInstance getOrCreateHazelcastInstance() {
    return Hazelcast.getOrCreateHazelcastInstance(getHazelcastConfig());
}

}

Why tcp configuration is not respecting interfaces configuration? On application startup with hibernate second level cache configuration

properties.setProperty("hibernate.cache.region.factory_class", "com.hazelcast.hibernate.HazelcastCacheRegionFactory");
properties.setProperty("hibernate.cache.hazelcast.use_lite_member", "true");
properties.setProperty("hibernate.cache.use_minimal_puts", "true");
properties.setProperty("hibernate.cache.hazelcast.use_native_client", "false");
properties.setProperty("hibernate.cache.hazelcast.instance_name", environment.getRequiredProperty("hazelcast.instanceName"));
properties.setProperty("hibernate.cache.hazelcast.native_client_address", "host.domain.eu:8080");
properties.setProperty("hibernate.cache.hazelcast.native_client_group", environment.getRequiredProperty("hazelcast.name"));
properties.setProperty("hibernate.cache.hazelcast.native_client_password", environment.getRequiredProperty("hazelcast.password"));

it prints log message with members from banned interfaces.

Members [1] {
    Member [192.168.216.122]:5701 this
}

Any hint?

After improvements sugested by A.Desai problem still exists with log:

13:58:45.086 [RMI TCP Connection(5)-127.0.0.1] INFO  o.s.o.j.LocalContainerEntityManagerFactoryBean - Building JPA container EntityManagerFactory for persistence unit 'default'
22-Jul-2016 13:58:46.903 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.hibernate.HazelcastCacheRegionFactory.null Starting up HazelcastCacheRegionFactory
22-Jul-2016 13:58:46.919 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.config.XmlConfigLocator.null Loading 'hazelcast-default.xml' from classpath.
22-Jul-2016 13:58:47.181 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.instance.DefaultAddressPicker.null [LOCAL] [dev] [3.6.4] Prefer IPv4 stack is true.
22-Jul-2016 13:58:47.191 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.instance.DefaultAddressPicker.null [LOCAL] [dev] [3.6.4] Picked Address[192.168.216.122]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
22-Jul-2016 13:58:47.220 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.system.null [192.168.216.122]:5701 [dev] [3.6.4] Hazelcast 3.6.4 (20160701 - 5b94d9f) starting at Address[192.168.216.122]:5701
22-Jul-2016 13:58:47.220 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.system.null [192.168.216.122]:5701 [dev] [3.6.4] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
22-Jul-2016 13:58:47.220 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.system.null [192.168.216.122]:5701 [dev] [3.6.4] Configured Hazelcast Serialization version : 1
22-Jul-2016 13:58:47.527 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.spi.OperationService.null [192.168.216.122]:5701 [dev] [3.6.4] Backpressure is disabled
22-Jul-2016 13:58:47.558 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.spi.impl.operationexecutor.classic.ClassicOperationExecutor.null [192.168.216.122]:5701 [dev] [3.6.4] Starting with 4 generic operation threads and 8 partition operation threads.
22-Jul-2016 13:58:48.180 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.instance.Node.null [192.168.216.122]:5701 [dev] [3.6.4] Creating MulticastJoiner
22-Jul-2016 13:58:48.184 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.core.LifecycleService.null [192.168.216.122]:5701 [dev] [3.6.4] Address[192.168.216.122]:5701 is STARTING
22-Jul-2016 13:58:48.293 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThreadingModel.null [192.168.216.122]:5701 [dev] [3.6.4] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads
22-Jul-2016 13:58:50.614 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.cluster.impl.MulticastJoiner.null [192.168.216.122]:5701 [dev] [3.6.4] 


Members [1] {
    Member [192.168.216.122]:5701 this
}

22-Jul-2016 13:58:50.681 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.core.LifecycleService.null [192.168.216.122]:5701 [dev] [3.6.4] Address[192.168.216.122]:5701 is STARTED
22-Jul-2016 13:58:53.465 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.hibernate.HazelcastCacheRegionFactory.null Starting up HazelcastCacheRegionFactory
22-Jul-2016 13:58:53.466 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.config.XmlConfigLocator.null Loading 'hazelcast-default.xml' from classpath.
22-Jul-2016 13:58:54.296 INFO [RMI TCP Connection(5)-127.0.0.1] com.hazelcast.partition.InternalPartitionService.null [192.168.216.122]:5701 [dev] [3.6.4] Initializing cluster partition table arrangement...

Solution: Moved configuration from java class to xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
  ~
  ~ Licensed under the Apache License, Version 2.0 (the "License");
  ~ you may not use this file except in compliance with the License.
  ~ You may obtain a copy of the License at
  ~
  ~ http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->

<!--
    The default Hazelcast configuration. This is used when no hazelcast.xml is present.
    Please see the schema for how to configure Hazelcast at https://hazelcast.com/schema/config/hazelcast-config-3.7.xsd
    or the documentation at https://hazelcast.org/documentation/
-->
<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.7.xsd"
           xmlns="http://www.hazelcast.com/schema/config"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <instance-name>node1</instance-name>
    <properties>
        <property name="hazelcast.initial.min.cluster.size">1</property>
        <property name="hazelcast.socket.bind.any">false</property>
        <property name="hazelcast.socket.server.bind.any">false</property>
    <property name="hazelcast.socket.client.bind">false</property>
    <property name="hazelcast.socket.client.bind.any">false</property>
    <property name="hazelcast.logging.type">slf4j</property>
    </properties>
    <group>
        <name>groupName</name>
        <password>password</password>
    </group>
    <network>
        <port auto-increment="true">5701</port>
        <reuse-address>true</reuse-address>
        <join>
            <multicast enabled="false"></multicast>
            <tcp-ip enabled="true">
                <interface>127.0.0.1</interface>
                <member-list>
                    <member>127.0.0.1:5701</member>
                </member-list>
            </tcp-ip>
            <aws enabled="false"></aws>
            <discovery-strategies>
            </discovery-strategies>
        </join>
        <ssl enabled="false"/>
        <socket-interceptor enabled="false"/>        
    </network>  

</hazelcast>

Radoslaw, did you have a chance to check my answer? – Vik Gamov Jul 26 '16 at 17:15 — Vik Gamov, Jul 26 '16 at 17:15

score 1 · Answer 1 · answered Jul 22 '16 at 11:37

Initially you are enabling the TCP-IP Config with joinConfig.getTcpIpConfig().setEnabled(true); but again you are overwriting it at the end with a new tcpIpConfig object. Please add this statement at the end to enable it again. I believe this should solve your problem else please post complete log statements.

tcpIpConfig.setEnabled(true);

    tcpIpConfig.setConnectionTimeoutSeconds(5);
    tcpIpConfig.setEnabled(true);
    joinConfig.setTcpIpConfig(tcpIpConfig);
    config.setNetworkConfig(networkConfig);

score 1 · Accepted Answer · answered Jul 25 '16 at 18:29

1

Let me quickly explain how interfaces are resolved in Hazelcast.

By default, Hazelcast will bind (accept incoming traffic) to all local network interfaces. If this is an unwanted behavior, you can set the hazelcast.socket.bind.any to false. In that case, Hazelcast will first use the interfaces configured in the interfaces/interfaces section of hazelcast.xml to resolve one interface to bind to. If none is found, Hazelcast will use the interfaces in the tcp-in/members to resolve one interface to bind to. If no interface is found, it will default to localhost.

Hope this helps!

Cheers,

Vik

answered Jul 25 '16 at 18:29

Vik Gamov

5,446
1
26
46

I have already used "hazelcast.socket.bind.any" with "false" value. That did not help. Problem was that I have badly configured reading hazelcast properties with spring framework. After removing java configuration from "HazelcastConfig" class above and moving configuration to XML file, which was passed via environment variable "hazelcast.config", everything worked like a charm. I am posting working XML with configured interfaces above. – Radosław Osiński Jul 28 '16 at 18:55
1

That's great news. Don't hesitate to ask more questions with #hazelcast tag – Vik Gamov Jul 28 '16 at 18:59

Hazelcast network interfaces

Any hint?

2 Answers2