0

This message is not as long as it seems - I only attached a lengthy log report at the end

Hi all,

I am running an experimental crawler implemented in Java. Recently, after a 36 hours operation, the JVM crashed issuing an exception access violation log file. I read a lot about this file, but it is still a little cryptic to me. Here is the relevant "Stack" version inside the generated log file:

Stack: [0x0000000033270000,0x0000000033370000],  sp=0x000000003336f8b0,  free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0x138998]
V  [jvm.dll+0x3f238a]
V  [jvm.dll+0x3efa30]
V  [jvm.dll+0x3ecfbe]
V  [jvm.dll+0x299f7a]
C  [msvcr100.dll+0x21d9f]
C  [msvcr100.dll+0x21e3b]
C  [kernel32.dll+0x159ed]
C  [ntdll.dll+0x2c541]

As you can see, at least from this log file, the error did not even start with some java code, but with the ntdll library, where instructions execution reached an exception at the jvm itself [Edit: or maybe the other way around? ended with the ntdll?]. I do not want to just dismiss this as "well... I'll reinstall the JVM and windows and maybe it'll work" (though I'd definitely do that), but try to see if there's something I can do about it. Some known issue or fixes that can be applied, other than the obvious ones I stated. The problem is this error occured after over 36 hours, so it's very hard to reproduce.

Also I need to say I am invoking

System.gc();

Every about 8 seconds to make sure memory is cleared. If that matters in anyway.\

I am not using any graphics libraries.

I am not using any JNI code.

Thanks a lot for reading and possibly helping,

Amir.

Attached - almost complete log file (hs_err_pid10544.log), without the very lengthy threads details:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000070ae8998, pid=10544, tid=10872
#
# JRE version: Java(TM) SE Runtime Environment (8.0_65-b17) (build 1.8.0_65-b17)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.65-b01 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# V  [jvm.dll+0x138998]
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x0000000002164800):  GCTaskThread [stack: 0x0000000033270000,0x0000000033370000] [id=10872]

siginfo: ExceptionCode=0xc0000005, reading address 0xffffffffffffffff

Registers:
RAX=0x0000000033730000, RBX=0x0000000000001991, RCX=0x0000000000000000, RDX=0x000000000000002a
RSP=0x000000003336f8b0, RBP=0x00000000021dd320, RSI=0x000000069fecdac4, RDI=0x000000028a46a950
R8 =0x007ffffffe052354, R9 =0x00000000021dd320, R10=0x000000000001c7f9, R11=0x0000000692210540
R12=0x000000069fecdbd0, R13=0x0000000000342e00, R14=0x000000069f1c23c0, R15=0x0000000000400000
RIP=0x0000000070ae8998, EFLAGS=0x0000000000010202

Top of Stack: (sp=0x000000003336f8b0)
0x000000003336f8b0:   000000003f7c4008 0000000070ac3166
0x000000003336f8c0:   0000000688823308 0000000070da1559
0x000000003336f8d0:   0000000000000006 0000000070bcb202
0x000000003336f8e0:   0000000000000000 0000000002164b60
0x000000003336f8f0:   0000000002159f60 0000000000342c00
0x000000003336f900:   0000000000001991 0000000070da238a
0x000000003336f910:   00000000021dd320 000000069f1c23c0
0x000000003336f920:   0000000070ff92b8 00b2ea6700001991
0x000000003336f930:   0acdfc080001c74b 0acdfc090001c74b
0x000000003336f940:   0acdb6a10001c70f 000000069f1c23c0
0x000000003336f950:   0000000600342c00 0000000070ae8a21
0x000000003336f960:   000000003eae9b00 0000000070bcb202
0x000000003336f970:   0000000002164800 0000000002159f60
0x000000003336f980:   0000000002164b70 0000000002164b60
0x000000003336f990:   0000000070ff92b8 0000000000000004
0x000000003336f9a0:   00000000666b8c60 0000000000000000 

Instructions: (pc=0x0000000070ae8998)
0x0000000070ae8978:   05 33 1c 6d 00 49 c1 e8 03 49 d3 e8 41 0f b6 c0
0x0000000070ae8988:   49 c1 e8 06 24 3f 0f b6 d0 48 8b 05 48 1c 6d 00
0x0000000070ae8998:   4a 8b 0c c0 48 0f a3 d1 72 3e 48 8b cf e8 16 0f
0x0000000070ae89a8:   fd ff 84 c0 74 32 48 8b d7 48 8b cd e8 e7 dd fc 


Register to memory mapping:

RAX=0x0000000033730000 is an unknown value
RBX=0x0000000000001991 is an unknown value
RCX=0x0000000000000000 is an unknown value
RDX=0x000000000000002a is an unknown value
RSP=0x000000003336f8b0 is an unknown value
RBP=0x00000000021dd320 is an unknown value
RSI=0x000000069fecdac4 is pointing into object: 0x000000069f1c23c0
[Ljava.util.HashMap$Node; 
 - klass: 'java/util/HashMap$Node'[]
 - length: 4194304
RDI=0x000000028a46a950 is pointing into the stack for thread: 0x000000003fdbb800
R8 =0x007ffffffe052354 is an unknown value
R9 =0x00000000021dd320 is an unknown value
R10=0x000000000001c7f9 is an unknown value
R11=0x0000000692210540 is an oop
java.util.HashMap$Node 
 - klass: 'java/util/HashMap$Node'
R12=0x000000069fecdbd0 is pointing into object: 0x000000069f1c23c0
[Ljava.util.HashMap$Node; 
 - klass: 'java/util/HashMap$Node'[]
 - length: 4194304
R13=0x0000000000342e00 is an unknown value
R14=0x000000069f1c23c0 is an oop
[Ljava.util.HashMap$Node; 
 - klass: 'java/util/HashMap$Node'[]
 - length: 4194304
R15=0x0000000000400000 is an unknown value


Stack: [0x0000000033270000,0x0000000033370000],  sp=0x000000003336f8b0,  free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0x138998]
V  [jvm.dll+0x3f238a]
V  [jvm.dll+0x3efa30]
V  [jvm.dll+0x3ecfbe]
V  [jvm.dll+0x299f7a]
C  [msvcr100.dll+0x21d9f]
C  [msvcr100.dll+0x21e3b]
C  [kernel32.dll+0x159ed]
C  [ntdll.dll+0x2c541]

... (Some code I didn't copy, contains a lot of threads descriptions) ...

Dynamic libraries:
0x000000013fd20000 - 0x000000013fd57000     C:\Program Files\Java\jdk1.8.0_65\bin\javaw.exe
0x00000000773f0000 - 0x0000000077599000     C:\Windows\SYSTEM32\ntdll.dll
0x00000000772d0000 - 0x00000000773ef000     C:\Windows\system32\kernel32.dll
0x000007fefd310000 - 0x000007fefd37c000     C:\Windows\system32\KERNELBASE.dll
0x000007fefee70000 - 0x000007fefef4b000     C:\Windows\system32\ADVAPI32.dll
0x000007fefdd00000 - 0x000007fefdd9f000     C:\Windows\system32\msvcrt.dll
0x000007fefee50000 - 0x000007fefee6f000     C:\Windows\SYSTEM32\sechost.dll
0x000007fefd690000 - 0x000007fefd7bd000     C:\Windows\system32\RPCRT4.dll
0x00000000771d0000 - 0x00000000772ca000     C:\Windows\system32\USER32.dll
0x000007feff600000 - 0x000007feff667000     C:\Windows\system32\GDI32.dll
0x000007feff670000 - 0x000007feff67e000     C:\Windows\system32\LPK.dll
0x000007fefd5c0000 - 0x000007fefd689000     C:\Windows\system32\USP10.dll
0x000007fefbc60000 - 0x000007fefbe54000     C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7601.17514_none_fa396087175ac9ac\COMCTL32.dll
0x000007fefdc80000 - 0x000007fefdcf1000     C:\Windows\system32\SHLWAPI.dll
0x000007feff350000 - 0x000007feff37e000     C:\Windows\system32\IMM32.DLL
0x000007fefeca0000 - 0x000007fefeda9000     C:\Windows\system32\MSCTF.dll
0x0000000074210000 - 0x00000000742e2000     C:\Program Files\Java\jdk1.8.0_65\jre\bin\msvcr100.dll
0x00000000709b0000 - 0x0000000071239000     C:\Program Files\Java\jdk1.8.0_65\jre\bin\server\jvm.dll
0x000007fee6bf0000 - 0x000007fee6bf9000     C:\Windows\system32\WSOCK32.dll
0x000007feff280000 - 0x000007feff2cd000     C:\Windows\system32\WS2_32.dll
0x000007feff190000 - 0x000007feff198000     C:\Windows\system32\NSI.dll
0x000007feedec0000 - 0x000007feedefb000     C:\Windows\system32\WINMM.dll
0x000007fefc2f0000 - 0x000007fefc2fc000     C:\Windows\system32\VERSION.dll
0x00000000775c0000 - 0x00000000775c7000     C:\Windows\system32\PSAPI.DLL
0x0000000074430000 - 0x000000007443f000     C:\Program Files\Java\jdk1.8.0_65\jre\bin\verify.dll
0x0000000074400000 - 0x0000000074429000     C:\Program Files\Java\jdk1.8.0_65\jre\bin\java.dll
0x00000000743e0000 - 0x00000000743f6000     C:\Program Files\Java\jdk1.8.0_65\jre\bin\zip.dll
0x000007fefdda0000 - 0x000007fefeb28000     C:\Windows\system32\SHELL32.dll
0x000007fefd7c0000 - 0x000007fefd9c3000     C:\Windows\system32\ole32.dll
0x000007fefd210000 - 0x000007fefd21f000     C:\Windows\system32\profapi.dll
0x00000000743c0000 - 0x00000000743da000     C:\Program Files\Java\jdk1.8.0_65\jre\bin\net.dll
0x000007fefc9e0000 - 0x000007fefca35000     C:\Windows\system32\mswsock.dll
0x000007fefc9d0000 - 0x000007fefc9d7000     C:\Windows\System32\wship6.dll
0x000007fefc3c0000 - 0x000007fefc3c7000     C:\Windows\System32\wshtcpip.dll
0x000007fefc820000 - 0x000007fefc87b000     C:\Windows\system32\DNSAPI.dll
0x000007fefafa0000 - 0x000007fefafc7000     C:\Windows\system32\IPHLPAPI.DLL
0x000007fefaf90000 - 0x000007fefaf9b000     C:\Windows\system32\WINNSI.DLL
0x000007fefa520000 - 0x000007fefa528000     C:\Windows\system32\rasadhlp.dll
0x000007fefae90000 - 0x000007fefaee3000     C:\Windows\System32\fwpuclnt.dll
0x0000000074390000 - 0x00000000743b4000     C:\Program Files\Java\jdk1.8.0_65\jre\bin\sunec.dll
0x000007fefca40000 - 0x000007fefca57000     C:\Windows\system32\CRYPTSP.dll
0x000007fefc700000 - 0x000007fefc747000     C:\Windows\system32\rsaenh.dll
0x000007fefd5a0000 - 0x000007fefd5be000     C:\Windows\system32\USERENV.dll
0x000007fefd0a0000 - 0x000007fefd0af000     C:\Windows\system32\CRYPTBASE.dll
0x000007fefada0000 - 0x000007fefadb1000     C:\Windows\system32\dhcpcsvc6.DLL
0x000007fefae20000 - 0x000007fefae38000     C:\Windows\system32\dhcpcsvc.DLL
0x0000000074370000 - 0x0000000074381000     C:\Program Files\Java\jdk1.8.0_65\jre\bin\nio.dll
0x000007fefb2b0000 - 0x000007fefb2c5000     C:\Windows\system32\NLAapi.dll
0x000007fee6930000 - 0x000007fee6945000     C:\Windows\system32\napinsp.dll
0x000007fee6910000 - 0x000007fee6929000     C:\Windows\system32\pnrpnsp.dll
0x000007fee6900000 - 0x000007fee690b000     C:\Windows\System32\winrnr.dll
0x0000000074360000 - 0x000000007436b000     C:\Program Files\Java\jdk1.8.0_65\jre\bin\sunmscapi.dll
0x000007fefd3f0000 - 0x000007fefd55c000     C:\Windows\system32\CRYPT32.dll
0x000007fefd200000 - 0x000007fefd20f000     C:\Windows\system32\MSASN1.dll
0x000007fefce40000 - 0x000007fefce4b000     C:\Windows\system32\Secur32.dll
0x000007fefd010000 - 0x000007fefd035000     C:\Windows\system32\SSPICLI.DLL
0x000007fefc600000 - 0x000007fefc60a000     C:\Windows\system32\credssp.dll
0x000007fefc970000 - 0x000007fefc9c2000     C:\Windows\system32\msv1_0.DLL
0x000007fefccf0000 - 0x000007fefcd04000     C:\Windows\system32\cryptdll.dll
0x0000000074350000 - 0x0000000074359000     C:\Program Files\Java\jdk1.8.0_65\jre\bin\w2k_lsa_auth.dll
0x000007fee43a0000 - 0x000007fee44c5000     C:\Windows\system32\dbghelp.dll

VM Arguments:
jvm_args: -Xmx5120m -Xss512m -Dfile.encoding=Cp1255 
java_command: main.Hub
java_class_path (initial): C:\Users\Me\workspace\Bot\bin;C:\Users\Me\workspace\avro-1.7.6.jar;C:\Users\Me\workspace\jsoup-1.7.3.jar;C:\Users\Me\workspace\jlangdetect-0.1.jar
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=C:/Program Files/Java/jre1.8.0_65/bin/server;C:/Program Files/Java/jre1.8.0_65/bin;C:/Program Files/Java/jre1.8.0_65/lib/amd64;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\Intel\OpenCL SDK\3.0\bin\x86;C:\Program Files (x86)\Intel\OpenCL SDK\3.0\bin\x64;C:\eclipse;
USERNAME=Me
OS=Windows_NT
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 60 Stepping 3, GenuineIntel



---------------  S Y S T E M  ---------------

OS: Windows 7 , 64 bit Build 7601 (6.1.7601.18409)

CPU:total 8 (4 cores per cpu, 2 threads per core) family 6 model 60 stepping 3, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, rtm, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2

Memory: 4k page, physical 16645332k(6688136k free), swap 33288800k(22549536k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.65-b01) for windows-amd64 JRE (1.8.0_65-b17), built on Oct  6 2015 16:39:20 by "java_re" with MS VC++ 10.0 (VS2010)

time: Sun Feb 07 07:52:26 2016
elapsed time: 134890 seconds (1d 13h 28m 10s)
amirkr
  • 173
  • 3
  • 15
  • 1
    It's unrelated, but you probably don't need to tell the jvm that you'd like to run GC. – Dave Newton Feb 07 '16 at 23:11
  • @DaveNewton Yes, you're right, the JVM usually does a very good job with the GC, but I've noticed that I have a lot of garbage buildup that is cleared by running the gc every few seconds, due to a large number of threads invoked and terminated. It reduces the memory usage by a large factor (as seen plainly by looking at the windows task manager's "memory usage" tab). Thanks for your input. – amirkr Feb 07 '16 at 23:13
  • The JVM will reduce its own memory usage as needed. You're also relying on a JVM implementation (granted, several run GC when you ask it to) but there's no guarantee that will always be the case. If you need memory, you need memory. – Dave Newton Feb 07 '16 at 23:29
  • I see. I'll test it again. – amirkr Feb 07 '16 at 23:34
  • Have you tried with different JVM versions? JVM bugs do happen, but often they pop up with one update and are fixed a few updates later. – the8472 Feb 08 '16 at 01:53
  • 1
    This could be a bug. I would make sure you have the latest JVM and windows update. – Peter Lawrey Feb 08 '16 at 03:12
  • The problem is I can't really count on reproducing the problem, since it doesn't always occur and if so, it happens after many hours. I've learned the hard way not to say it's the JVM's fault because it's usually mine's - I'm not as experienced as the JVM developers. But It seems I'd have to resort to blaming them this time and hope the latest versions, after a complete format of the machine, would help. – amirkr Feb 08 '16 at 10:33
  • This sort of thing can sometimes indicate a defective RAM chip somewhere. Do you have any other applications on the same machine which have experienced random crashes? Or do you find that this happens with this application on a specific machine? – sisyphus Feb 08 '16 at 11:38
  • Well, when you go through the specification, you won’t find any scenario where the spec says that such a crash was a legal response for a JVM. Only when you’re not in the JVM, i.e. executing native code or poking around with `Unsafe`, you may be responsible. So having a crash with ordinary Java code, is a JVM bug, unless the operating system or a hardware fault are to blame. – Holger Feb 08 '16 at 13:22
  • Thanks for all your comments. I will run memtest to see if there's any hardware memory issue. – amirkr Feb 09 '16 at 07:07

0 Answers0