1

I am running into this error on trying to load a Avro file (size 134 KB).My pom dependencies are below. I am creating this Avro from a protobuf message which works fine.

pom dependencies :

<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.7.7</version>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro-protobuf</artifactId>
<version>1.7.7</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.0.0</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-avro_2.11</artifactId>
<version>3.0.0</version>
</dependency>

Exception :

Exception in thread "main" java.lang.StackOverflowError
    at scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
    at scala.collection.Iterator$class.exists(Iterator.scala:919)
    at scala.collection.AbstractIterator.exists(Iterator.scala:1336)
    at scala.collection.IterableLike$class.exists(IterableLike.scala:77)
    at scala.collection.AbstractIterable.exists(Iterable.scala:54)
    at com.databricks.spark.avro.SchemaConverters$.toSqlType(SchemaConverters.scala:75)
    at com.databricks.spark.avro.SchemaConverters$$anonfun$1.apply(SchemaConverters.scala:56)
    at com.databricks.spark.avro.SchemaConverters$$anonfun$1.apply(SchemaConverters.scala:55)
ρяσѕρєя K
  • 132,198
  • 53
  • 198
  • 213
Nitin Kumar
  • 219
  • 2
  • 10
  • Perhaps this is a late response, but there might be 2 possible schema-related reasons: 1) having too many nested levels in your avro schema (usually something about 5 levels should be quite fine), 2) having recursive fields in the schema. For me it was the latter issue and was fixed by renaming the fields. – Eb Abadi Oct 18 '18 at 18:39

0 Answers0