7

I was wondering if some of the JVM gurus out there can briefly explain the following error. What does it actually mean in technical terms and what are the sequences of events that can lead to this error?

java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
wFateem
  • 346
  • 1
  • 2
  • 11
  • How did *you* get the error in the first place? – Lino Aug 06 '17 at 19:58
  • Have you tried Googling the error message? There are some interesting results that might explain it, like https://gist.github.com/rednaxelafx/1372200/58a95fbf57be0930ceb448280f84e22217cfa37f in which someone looked at the C code of the JVM and found something about a SIGBUS error. – ajb Aug 06 '17 at 20:12
  • 1
    I didn't personally run into this error, but I have been looking through some bugs reported in other software projects, like Kafka for instance and saw this particular error pop-up during different scenarios with different stacks. I was just curious about the error. – wFateem Aug 06 '17 at 23:11
  • 2
    Check out for disk errors (sudo dmesg -T) – mac Aug 18 '18 at 13:40

3 Answers3

16

This error means that sun.misc.Unsafe.getX() or putX() memory access resulted in SIGBUS error, which was then caught by JVM and translated to asynchronous InternalError.

A bit more details:

  • sun.misc.Unsafe is JDK private API that allows to access native memory directly from Java. This API is a foundation for Direct ByteBuffers and particularly MappedByteBuffers.

  • In certain cases an access to memory-mapped region of a file may lead to OS-level exception, namely SIGBUS. Typical examples are:

    1. A memory-mapped buffer is accessed after the underlying file has been truncated.
    2. A file on a network drive has been mapped to memory, and the mapped buffer is accessed after the network connection has been lost.
    3. An attempt to write to a page mapped to a file on tmpfs filesystem results in out-of-memory (by default tmpfs space is limited by 50% of total RAM).

  • HotSpot JVM cannot efficiently detect these problems beforehand. It compiles Unsafe.getX / putX calls to a simple memory access instruction. Additional checks to see if a memory region is valid would be too expensive.

  • Instead JVM handles SIGBUG signal. If it sees the error has happened in Unsafe call, it posts InternalError to current thread and continues execution.
  • IOException would be more appropriate, but JVM cannot throw it or any other exception, since ByteBuffer public contract does not allow its get/put methods to throw any exception.
  • If Unsafe memory access has failed in JIT-compiled method, JVM does not throw an exception immediately (again, it would be too expensive for such hot ByteBuffer API). Instead it posts asynchronous InternalError to the current thread. It means the error would be actually thrown at the nearest native method or at the nearest call to VM runtime. Hence the word "recent" in the error message.

See JDK-4454115 comments describing the solution for the related JDK bug.

apangin
  • 92,924
  • 10
  • 193
  • 247
4

This can also happen due to disk space issues.

z atef
  • 7,138
  • 3
  • 55
  • 50
2

This error may be induced by corrupted index which in turn becomes corrupted due to java.io.IOException: No space left on device

Look into your_kafka_home/logs/server.log

It all starts like this:

[2019-11-22 17:27:25,204] FATAL [Replica Manager on Broker 1]: Halting due to unrecoverable I/O error while handling produce request:  (kafka.server.ReplicaManager)
kafka.common.KafkaStorageException: I/O exception in append to log 'xxx-response-1-0'
    at kafka.log.Log.append(Log.scala:443)
    at kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:451)
Caused by: java.io.IOException: No space left on device
    at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
    at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
    at sun.nio.ch.IOUtil.write(IOUtil.java:65)
    at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
    at org.apache.kafka.common.record.MemoryRecords.writeFullyTo(MemoryRecords.java:82)
    at org.apache.kafka.common.record.FileRecords.append(FileRecords.java:159)
    at kafka.log.LogSegment.append(LogSegment.scala:110)
    at kafka.log.Log.append(Log.scala:429)
    ... 21 more

is followed by (possibly mutilple times):

[2019-11-22 17:28:00,395] WARN Found a corrupted index file due to requirement
failed: Corrupt index found, index file (/opt/kafka/kafka-logs/__consumer_offsets- 
9/00000000000000000000.index) has non-zero size but the last offset is 0 which is no
larger than the base offset 0.}. deleting /opt/kafka/kafka-logs/__consumer_offsets-9
/00000000000000000000.timeindex, /opt/kafka/kafka-logs/__consumer_offsets-9
/00000000000000000000.index and rebuilding index... (kafka.log.Log)

and finally comes to :

[2019-11-22 17:28:02,025] FATAL [Kafka Server 1], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.InternalError: **a fault occurred in a recent unsafe memory access operation in compiled Java code**
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
    at kafka.log.TimeIndex.maybeAppend(TimeIndex.scala:107)
    at kafka.log.LogSegment.$anonfun$recover$1(LogSegment.scala:238
Сергей Кондурушкин
  • 21
  • 3