Hazelcast adding address to blacklist and client not connecting

Question

I am new to Hazelcast and in my application when I am starting Hazelcast member it is showing two addresses been blacklisted don't know why, what is the best configuration for Hazelcast clustering?

Here is the log that I am getting:

2021-07-22 13:15:51.823  INFO 212124 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8081 (http)
2021-07-22 13:15:51.844  INFO 212124 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2021-07-22 13:15:51.845  INFO 212124 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.44]
2021-07-22 13:15:52.094  INFO 212124 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2021-07-22 13:15:52.094  INFO 212124 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 4676 ms
2021-07-22 13:15:52.417  INFO 212124 --- [           main] com.hazelcast.instance.AddressPicker     : [LOCAL] [APP] [4.2] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [192.168.0.104]
2021-07-22 13:15:53.350  INFO 212124 --- [           main] com.hazelcast.system                     : [192.168.0.104]:5701 [APP] [4.2] Hazelcast 4.2 (20210324 - 405cfd1) starting at [192.168.0.104]:5701
2021-07-22 13:15:54.178  INFO 212124 --- [           main] com.hazelcast.instance.impl.Node         : [192.168.0.104]:5701 [APP] [4.2] Using TCP/IP discovery
2021-07-22 13:15:54.180  WARN 212124 --- [           main] com.hazelcast.cp.CPSubsystem             : [192.168.0.104]:5701 [APP] [4.2] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
2021-07-22 13:15:54.680  INFO 212124 --- [           main] c.h.internal.diagnostics.Diagnostics     : [192.168.0.104]:5701 [APP] [4.2] Diagnostics started
2021-07-22 13:15:54.680  INFO 212124 --- [           main] c.h.i.diagnostics.BuildInfoPlugin        : [192.168.0.104]:5701 [APP] [4.2] Plugin:active
2021-07-22 13:15:54.681  INFO 212124 --- [           main] c.h.i.d.SystemPropertiesPlugin           : [192.168.0.104]:5701 [APP] [4.2] Plugin:active
2021-07-22 13:15:54.681  INFO 212124 --- [           main] c.h.i.d.ConfigPropertiesPlugin           : [192.168.0.104]:5701 [APP] [4.2] Plugin:active
2021-07-22 13:15:54.684  INFO 212124 --- [           main] c.h.i.d.PendingInvocationsPlugin         : [192.168.0.104]:5701 [APP] [4.2] Plugin:active: period-millis:30000 threshold:1
2021-07-22 13:15:54.686  INFO 212124 --- [           main] c.h.internal.diagnostics.MetricsPlugin   : [192.168.0.104]:5701 [APP] [4.2] Plugin:active, period-millis:60000
2021-07-22 13:15:54.686  INFO 212124 --- [           main] c.h.i.diagnostics.SlowOperationPlugin    : [192.168.0.104]:5701 [APP] [4.2] Plugin:active, period-millis:30000
2021-07-22 13:15:54.687  INFO 212124 --- [           main] c.h.i.d.PendingInvocationsPlugin         : [192.168.0.104]:5701 [APP] [4.2] Plugin:active: period-millis:30000 threshold-millis:5000
2021-07-22 13:15:54.687  INFO 212124 --- [           main] c.h.i.d.PendingInvocationsPlugin         : [192.168.0.104]:5701 [APP] [4.2] Plugin:active: period-millis:5000
2021-07-22 13:15:54.688  INFO 212124 --- [           main] c.h.i.d.OperationProfilerPlugin          : [192.168.0.104]:5701 [APP] [4.2] Plugin:active, period-millis:5000
2021-07-22 13:15:54.688  INFO 212124 --- [           main] .h.i.d.MemberHazelcastInstanceInfoPlugin : [192.168.0.104]:5701 [APP] [4.2] Plugin:active, period-millis:60000
2021-07-22 13:15:54.689  INFO 212124 --- [           main] c.h.i.diagnostics.SystemLogPlugin        : [192.168.0.104]:5701 [APP] [4.2] Plugin:active: logPartitions:false
2021-07-22 13:15:54.692  INFO 212124 --- [SchedulerThread] c.h.internal.diagnostics.Diagnostics     : [192.168.0.104]:5701 [APP] [4.2] Diagnostics log directory is [D:\Parth\Hazelcast\logs]
2021-07-22 13:15:54.694  INFO 212124 --- [           main] c.h.i.diagnostics.StoreLatencyPlugin     : [192.168.0.104]:5701 [APP] [4.2] Plugin:active: period-millis:60000 resetPeriod-millis:0
2021-07-22 13:15:54.696  INFO 212124 --- [           main] .h.i.d.MemberHazelcastInstanceInfoPlugin : [192.168.0.104]:5701 [APP] [4.2] Plugin:active, period-millis:10000
2021-07-22 13:15:54.697  INFO 212124 --- [           main] c.h.i.d.OperationHeartbeatPlugin         : [192.168.0.104]:5701 [APP] [4.2] Plugin:active: period-millis:10000 max-deviation:33%
2021-07-22 13:15:54.699  INFO 212124 --- [           main] com.hazelcast.core.LifecycleService      : [192.168.0.104]:5701 [APP] [4.2] [192.168.0.104]:5701 is STARTING
2021-07-22 13:15:59.367  INFO 212124 --- [cached.thread-3] c.h.internal.cluster.impl.TcpIpJoiner    : [192.168.0.104]:5701 [APP] [4.2] [192.168.0.104]:5702 is added to the blacklist.
2021-07-22 13:15:59.524  INFO 212124 --- [cached.thread-5] c.h.internal.cluster.impl.TcpIpJoiner    : [192.168.0.104]:5701 [APP] [4.2] [192.168.0.104]:5703 is added to the blacklist.
2021-07-22 13:16:00.178  INFO 212124 --- [           main] c.h.internal.cluster.ClusterService      : [192.168.0.104]:5701 [APP] [4.2] 

Members {size:1, ver:1} [
    Member [192.168.0.104]:5701 - c35ebca4-ca5f-404c-ba0e-1c5e1287aa21 this
]

2021-07-22 13:16:00.194  INFO 212124 --- [           main] com.hazelcast.core.LifecycleService      : [192.168.0.104]:5701 [APP] [4.2] [192.168.0.104]:5701 is STARTED
2021-07-22 13:16:00.514  INFO 212124 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Initializing ExecutorService 'applicationTaskExecutor'
2021-07-22 13:16:01.489  INFO 212124 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8081 (http) with context path ''
2021-07-22 13:16:01.506  INFO 212124 --- [           main] com.APPhub.Application                   : Started Application in 15.086 seconds (JVM running for 15.996)

Here two addresses getting blacklisted not sure why.

And my server config is as below:

        Config config = new Config();
        
        config.getNetworkConfig().setPort(5701)
        .setPortAutoIncrement( true ).setPortCount(20);
        
        config.getNetworkConfig().setReuseAddress( true );
        

        System.setProperty( "hazelcast.diagnostics.enabled", "true" );
        System.setProperty( "hazelcast.diagnostics.metric.level", "info" );
        System.setProperty( "hazelcast.diagnostics.invocation.sample.period.seconds", "30" );
        System.setProperty( "hazelcast.diagnostics.pending.invocations.period.seconds", "30" );
        System.setProperty( "hazelcast.diagnostics.slowoperations.period.seconds", "30" );
        System.setProperty( "hazelcast.diagnostics.storeLatency.period.seconds", "60" );
        System.setProperty( "hazelcast.diagnostics.directory", "D:/Parth/Hazelcast/logs/" );
        System.setProperty( "hazelcast.diagnostics.filename.prefix", "Server" );
        
        
        config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
        
        config.getNetworkConfig().getInterfaces().addInterface("127.0.0.1");
        
        JoinConfig join = config.getNetworkConfig().getJoin();
        join.getTcpIpConfig().addMember("192.168.0.104");
        join.getTcpIpConfig().setEnabled(true);

When trying to connect client it gives:

2021-07-22 12:07:24.081  INFO 192584 --- [ration.thread-0] c.h.c.i.p.t.AuthenticationMessageTask    : [192.168.0.104]:5701 [APP] [4.2] Received auth from Connection[id=2, /127.0.0.1:5701->/127.0.0.1:5703, qualifier=null, endpoint=[127.0.0.1]:5703, alive=true, connectionType=JVM, planeIndex=-1], successfully authenticated, clientUuid: 6bebb36c-b865-4fd5-aa12-d0f3ec82ae8d, client version: 4.2

When Connect 1 ==============================================================
6bebb36c-b865-4fd5-aa12-d0f3ec82ae8d
8c54b788-7dfb-401a-ae2c-70444534fa8d
When Connect 1 ==============================================================
2021-07-22 12:07:38.902  WARN 192584 --- [.IO.thread-in-1] c.h.i.server.tcp.TcpServerConnection     : [192.168.0.104]:5701 [APP] [4.2] Connection[id=1, /127.0.0.1:5701->/127.0.0.1:5702, qualifier=null, endpoint=[127.0.0.1]:5702, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Exception in Connection[id=1, /127.0.0.1:5701->/127.0.0.1:5702, qualifier=null, endpoint=[127.0.0.1]:5702, alive=true, connectionType=JVM, planeIndex=-1], thread=hz.APP1.IO.thread-in-1

java.io.IOException: An existing connection was forcibly closed by the remote host
    at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[na:1.8.0_275]
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) ~[na:1.8.0_275]
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_275]
    at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_275]
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[na:1.8.0_275]
    at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:119) ~[hazelcast-all-4.2.jar:4.2]
    at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383) [hazelcast-all-4.2.jar:4.2]
    at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368) [hazelcast-all-4.2.jar:4.2]
    at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294) [hazelcast-all-4.2.jar:4.2]
    at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249) [hazelcast-all-4.2.jar:4.2]

Client Configuration:

ClientConfig clientConfig = new ClientConfig();
        clientConfig.setClusterName("APP");
        clientConfig.getNetworkConfig().addAddress("localhost");
        clientConfig.getNetworkConfig().setSmartRouting(true);
        clientConfig.getNetworkConfig().addOutboundPortDefinition(5702-5720);
        
        
        clientConfig.setProperty("hazelcast.client.heartbeat.timeout", "60000");
        clientConfig.setProperty("hazelcast.client.heartbeat.timeout", "5000");
        
        ClientConnectionStrategyConfig connectionStrategyConfig = clientConfig.getConnectionStrategyConfig();
        connectionStrategyConfig.setReconnectMode(ClientConnectionStrategyConfig.ReconnectMode.ON);
        connectionRetryConfig.setInitialBackoffMillis(1000)
                             .setMaxBackoffMillis(60000)
                             .setMultiplier(2)
                             .setClusterConnectTimeoutMillis(1000)
                             .setJitter(0.2);

Let me know what wrong am I doing or what is best configuration for Member and client clustering where Member will be 1 and clients will be multiple.

As far as I know, you are not doing anything wrong, and you can ignore the blacklist message. The cluster will still be formed if you add more nodes with the same configuration on 192.168.0.104 machine, because new members will also perform scan and find the first node. Also, cause you are new to hazelcast, I'd recommend you to look at clusterName configuration, it's the easiest way to avoid unexpected members in clusters. — Vitaly Roslov, Jul 22 '21 at 10:01
@vitalyros Thank you for the replay but There is something wrong as when I am connecting two clients with cluster member one of the client is getting disconnected and showing: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection, regarding clutername I don't think it is a problem as I have only One Member and multiple clients so clustername is same for all. — user3458271, Jul 22 '21 at 10:46
Does the cluster node log anything when the second client connects? My hypothesis is windows firewall or network configurations being the problem. — Vitaly Roslov, Jul 22 '21 at 11:35
No not much logs it just disconnect by itself with this let me try turn off firewall: com.hazelcast.spi.exception.TargetDisconnectedException: Heartbeat timed out to connection ClientConnection{alive=true, connectionId=1, channel=NioChannel{/127.0.0.1:5703->localhost/127.0.0.1:5701}, remoteAddress=[localhost]:5701, lastReadTime=2021-07-22 17:04:29.122, lastWriteTime=2021-07-22 17:04:37.712, closedTime=never, connected server version=4.2} — user3458271, Jul 22 '21 at 11:39

score 1 · Answer 1 · answered Jul 22 '21 at 12:13

1

It's likely that the heartbeat timeout for the client is too short.

You've probably copied the heartbeat configuration from https://docs.hazelcast.com/imdg/4.2/clients/java.html

clientConfig.setProperty("hazelcast.client.heartbeat.timeout", "60000");
clientConfig.setProperty("hazelcast.client.heartbeat.interval", "5000");

but for the second line you've set timeout again instead of interval, overriding the value of 60000 with 5000

answered Jul 22 '21 at 12:13

Vitaly Roslov

313
1
8

OOhhh what yes , thank you man it's connecting now, so stupid of me bad mistake, but let me try it few times. – user3458271 Jul 22 '21 at 12:50

Hazelcast adding address to blacklist and client not connecting

1 Answers1