2

I am using Spark3.1.1/Hadoop3.2.2 on Solaris 11. When using Spark to write dataframe to Hadoop in parquete format, JVM crashed with SIGBUS error (Java Frame: sun.misc.UnSafe.GetInt). Error happens only when running in Solaris with SPARC 64 bit, not on other platforms such as Centos or Ubuntu. I don't know if there are problems with Spark 3.x on SPARC platform or not? Anybody has any idea? JVM crashed on Spark writing parquet

Andrew Henle
  • 32,625
  • 3
  • 24
  • 56
Duc Duong
  • 21
  • 3
  • 1
    *I don't know if there are problems with Spark 3.x on SPARC platform or not?* Almost certainly a bug in the code - it's likely violating strict aliasing and using an unaligned address. Both of those invoke undefined behavior, but x86 systems mostly let crap code like that go without errors. Other architectures like SPARC and ARM don't - they have actual alignment restrictions and if your code violates those restrictions you get `SIGBUS`. If you can post a full stack trace of the native code, and somehow examine the source code, you might be able to figure out what the bug is. – Andrew Henle Feb 03 '22 at 01:43
  • I found that not only Spark 3.x but even 2.x or 1.x caused that error. And not only in dataframe writing but in some other function. The source code is very simple, just a dataframe saving or a model training command or everything that calls to org.apache.Unsafe.Platform package. – Duc Duong Feb 06 '22 at 08:38
  • V [libjvm.so+0xd716c8] Unsafe_GetLong+0x174 j sun.misc.Unsafe.getLong(Ljava/lang/Object;J)J+12 j org.apache.spark.unsafe.Platform.getLong(Ljava/lang/Object;J)J+5 j org.apache.spark.unsafe.bitset.BitSetMethods.isSet(Ljava/lang/Object;JI)Z+66 j org.apache.spark.sql.catalyst.expressions.UnsafeRow.isNullAt(I)Z+14 j org.apache.spark.sql.catalyst.expressions.JoinedRow.isNullAt(I)Z+16 org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificMutableProjection.apply(Ljava/lang/Object;)Ljava/lang/Object;+14 jorg.apache.spark.sql.execution.aggregate.TungstenAggregationIterator – Duc Duong Feb 06 '22 at 08:42
  • Above is part of error log – Duc Duong Feb 06 '22 at 08:44
  • How old is your JVM? – Andrew Henle Feb 06 '22 at 12:37
  • The Java version is 1.8 – Duc Duong Feb 07 '22 at 06:54
  • So it's Java 8, version 1, from 2014? – Andrew Henle Feb 07 '22 at 11:47
  • How to know the year? Just this information: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode solaris-sparc compressed oops) – Duc Duong Feb 08 '22 at 15:50

0 Answers0