I'm facing a bug using the logback on GCP
we have 30 number java programs and which are same version code running inside the GKE pod, it is been observed the case around 12 number JVM hangs becomes unresponsive at the same time , but doesn't meet this case every day.
Thread dump analysis confirm them hangs with console logging.
"Calc-10068-RT-ZNA" #21 daemon prio=5 os_prio=0 cpu=9811.37ms elapsed=19646.04s tid=0x00007f3f888d6000 nid=0x90a runnable [0x00007f3f5cfdf000]
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(java.base@11.0.16/Native Method)
at java.io.FileOutputStream.write(java.base@11.0.16/Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(java.base@11.0.16/Unknown Source)
at java.io.BufferedOutputStream.flush(java.base@11.0.16/Unknown Source)
- locked <0x0000000680032ab0> (a java.io.BufferedOutputStream)
at java.io.PrintStream.write(java.base@11.0.16/Unknown Source)
- locked <0x0000000680032a88> (a java.io.PrintStream)
at java.io.FilterOutputStream.write(java.base@11.0.16/Unknown Source)
at ch.qos.logback.core.joran.spi.ConsoleTarget$1.write(ConsoleTarget.java:37)
at ch.qos.logback.core.OutputStreamAppender.writeBytes(OutputStreamAppender.java:199)
at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:231)
at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
at ch.qos.logback.classic.Logger.info(Logger.java:579)
at com.xxx.fbidx.formula.formulaGroup.BaseFormulaGroup.run(BaseFormulaGroup.java:411)
at com.xxx.fbidx.calcengine.FbidxCalcEngineApps$FbidxCalcTask.run(FbidxCalcEngineApps.java:238)
at com.xxx.fbidx.common.scheduler.FixedRateTimerScheduler$TimerSchedulerTimerTask.run(FixedRateTimerScheduler.java:48)
- locked <0x00000006803fb910> (a java.util.ArrayList)
at java.util.TimerThread.mainLoop(java.base@11.0.16/Unknown Source)
at java.util.TimerThread.run(java.base@11.0.16/Unknown Source)
Environment details below
Kernel version: 5.10.162+
OS image: Container-Optimized OS from Google
Container runtime version: docker://20.10.6
kubelet version: v1.22.17-gke.3100
kube-proxy version: v1.22.17-gke.3100
Logback Version: 1.2.10
Slf4J Version : 1.7.30
Please let me know if u have idea , thanks!!
