Kafka Streams - Application JVMs crashing randomly

Question

I have a Kafka Streams application running on multiple JVMs (5 in total) to improve throughput. It runs fine for an hour or so and then, each JVM starts crashing one after another except the last one. is crashing randomly without exceptions in my code. When I looked at the core dump, it says:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f939eb57255, pid=9669, tid=0x00007f93883f3700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x5c5255]  G1ParScanThreadState::copy_to_survivor_space(InCSetState, oopDesc*, markOopDesc*)+0x45

I'm unable to correlate this error with my code.

Looks like a bug in the G1 collector. Can you try the latest update of Java 8? — Peter Lawrey, Nov 05 '18 at 18:08
If that doesn't help, I would try the CMS collector instead. — Peter Lawrey, Nov 05 '18 at 18:10
Btw Chronicle Queue can probably handle your load with one thread and is almost GC free (I wrote it) — Peter Lawrey, Nov 05 '18 at 19:18

score 0 · Answer 1 · answered May 02 '20 at 17:25

0

Based on the comments, it would be best to use the latest Java version that Kafka supports, for example, Java 11 (and optionally with Scala 2.13).

Overwise, you can change the Garbage Collector to something like CMS instead of G1GC

answered May 02 '20 at 17:25

OneCricketeer

179,855
19
132
245

Kafka Streams - Application JVMs crashing randomly

1 Answers1