0

NOTE: Found the root cause in application code using hazelcast which started to execute after 15 min, the code retrieved almost entire data, so the issue NOT in hazelcast, leaving the question here if anyone will see same side effect or wrong code.

What can cause heavy traffic between Hazelcast (v3.12.12, also tried 4.1.1) 2 nodes ?

It holds maps with lot of data, no new entries are added/removed within that time, only map values are updated.

Java 11, Memory usage 1.5GB out of 12GB, no full GCs identified.

Following JFR the high IO is from: com.hazelcast.internal.networking.nio.NioThread.processTaskQueue() enter image description here

Below chart of Network IO, 15 min after start traffic jumps from 15 to 60 MB. From application perspective nothing changed after these 15 min.

enter image description here

Pavel
  • 33
  • 1
  • 5

1 Answers1

0

This smells garbage collection, you are most likely to be running into long gc pauses. Check your gc logs, which you can enable using verbose gc settings on all members. If there are back-to-back GCs then you should do various things:

  1. increase the heap size
  2. tune your gc, I'd look into G1 (with -XX:MaxGCPauseMillis set to a reasonable number) and/or ZGC.
wildnez
  • 1,078
  • 1
  • 6
  • 10
  • There is no issues with GC, 9 GB heap free. Running JFR the high IO coming from com.hazelcast.internal.networking.nio.NioThread.processTaskQueue(), updated my question with last details. – Pavel May 22 '22 at 08:57