2

I have a web-app running on JBoss that executes various http requests with Apache http client library version 4.5.1. I had experienced various jvm crashes because of an EXCEPTION_ACCESS_VIOLATION that happens randomly and generates hs_err_pidXXXX.log and hs_err_pidXXXX.mdmp dump.

Here's part of the log file:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000005a2364d, pid=2032, tid=3168
#
# JRE version: Java(TM) SE Runtime Environment (7.0_51-b13) (build 1.7.0_51-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.51-b03 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# J  org.apache.http.client.protocol.RequestAddCookies.process(Lorg/apache/http/HttpRequest;Lorg/apache/http/protocol/HttpContext;)V
#
# Core dump written. Default location: C:\jboss-as-7.1.1\bin\hs_err_pid2032.mdmp
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x0000000011cb3000):  JavaThread "foo_QuartzScheduler_Worker-1" [_thread_in_Java, id=3168, stack(0x0000000010c20000,0x0000000010d20000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x00000003803aba48

Registers:
RAX=0x0000000776011fa0, RBX=0x0000000000000004, RCX=0x0000000000000000, RDX=0x0000000000000001
RSP=0x0000000010d1d270, RBP=0x0000000000000000, RSI=0x0000000775f54a38, RDI=0x0000000775f54a34
R8 =0x000000000000000a, R9 =0x0000000000000000, R10=0x0000000000000001, R11=0x0000000070075748
R12=0x0000000000000000, R13=0x0000000000000001, R14=0x00000007dd1e0988, R15=0x0000000011cb3000
RIP=0x0000000005a2364d, EFLAGS=0x0000000000010206

Top of Stack: (sp=0x0000000010d1d270)
0x0000000010d1d270:   01d2418502f38030 0000000776459470
0x0000000010d1d280:   00000007e5d061a8 0000000010d1d300
0x0000000010d1d290:   0000000700000005 00000007e5d060e8
0x0000000010d1d2a0:   00000007e5d067a0 00000005ffffffff
0x0000000010d1d2b0:   00000007e5d066a8 00000007dcde5c88
0x0000000010d1d2c0:   00000007dcde5c88 00000001fcba194b
0x0000000010d1d2d0:   0000000770075748 0000002300000000
0x0000000010d1d2e0:   00000007dd1e07f0 00000007dcde5c88
0x0000000010d1d2f0:   00000007e5d0ca58 00000022fcba0cd9
0x0000000010d1d300:   000000000000003f 00000000ffffffff
0x0000000010d1d310:   eebbac9adc1ccc08 00000007e5d062d8
0x0000000010d1d320:   00000007e5d065c8 00000000046d31a4
0x0000000010d1d330:   00000007a2ee3370 ffffffff0310c6fc
0x0000000010d1d340:   0000000000000006 0000000002d7b170
0x0000000010d1d350:   00000007e5d060e8 00000007e5d062d8
0x0000000010d1d360:   00000007dc1cd5a0 0000000700000009

I studied the log header format from Oracle documentation and I searched the meaning of it on the web.

I found out from here that "J" frame type crashes occurr when there's a bug with the compiler that has built the code specified in the log's message (in this case org.apache.http.client.protocol.RequestAddCookies.process(Lorg/apache/http/HttpRequest;Lorg/apache/http/protocol/HttpContext;)V ).

I checked http client jar's manifest and I saw that it was built with jdk 1.7.0_75.

My solution is to download the source code of the library and then compile it again with a more recent compiler (e.g. 1.7.0_80 ). If my theory is correct, then JVM shouldn't crash anymore.

Unfortunately the bug is not reproducible, so I can't test it easily.

I would like to know if my reasoning is correct. Thanks in advance.

Svech87
  • 88
  • 1
  • 1
  • 7
  • there are many google-results for https://www.google.de/?gws_rd=ssl#q=JavaThread+QuartzScheduler_Worker And many of them report problems with the jre. Could you update the jre? – Tobias Otto Jan 12 '17 at 16:49
  • @Tobias, thanks for helping. Firstly I tried running the app with 1.7.0_55 then I moved to 1.7.0_75 but the bug still occurred. I read some of the result you provided, but It seems that they have different frame type: they reported "V" frame type, which means VM frame. I have the "J" type, which is related to compiled Java frames. – Svech87 Jan 12 '17 at 19:39

1 Answers1

0

You got the crash here in "foo_QuartzScheduler_Worker-1" which is your application thread, secondly crash frame doesn't show any reference to JDK frames,

Problematic frame:

J org.apache.http.client.protocol.RequestAddCookies.process(Lorg/apache/http/HttpRequest;Lorg/apache/http/protocol/HttpContext;)V

# It doesn't look like java issue, to be very sure you need to provide the complete crash logs

Fairoz
  • 1,616
  • 13
  • 16
  • Thanks, Fairoz. Maybe I misunderstand the meaning of "Java frame type". So you say that these kind of errors occur only on frames related to classes in the java package? [This tutorial](http://ksoong.org/troubleshooting/crash/jvm-crash-cases.html) claims that "compiler bug generate some bad compiled java code to cause your crash". If it's true, it means that any java code can cause this kind of crash, even my application code (in the example there's even the very same class and method of mine). Anyway, I'll post the entire log file. – Svech87 Jan 16 '17 at 08:01