Spark SQL Stackoverflow

Question

I'm a newbie on spark and spark sql and I was trying to make the example that is on Spark SQL website, just a simple SQL query after loading the schema and data from a JSON files directory, like this:

import sqlContext.createSchemaRDD
val sqlContext = new org.apache.spark.sql.SQLContext(sc)
val path = "/home/shaza90/Desktop/tweets_1428981780000"
val tweet = sqlContext.jsonFile(path).cache()

tweet.registerTempTable("tweet")
tweet.printSchema() //This one works fine


val texts = sqlContext.sql("SELECT tweet.text FROM tweet").collect().foreach(println) 

The exception that I'm getting is this one:

java.lang.StackOverflowError

    at scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)

Update

I'm able to execute select * from tweet but whenever I use a column name instead of * I get the error.

Any Advice?

Answer 1

This is SPARK-5009 and has been fixed in Apache Spark 1.3.0.

The issue was that to recognize keywords (like SELECT) with any case, all possible uppercase/lowercase combinations (like seLeCT) were generated in a recursive function. This recursion would lead to the StackOverflowError you're seeing, if the keyword was long enough and the stack size small enough. (This suggests that if upgrading to Apache Spark 1.3.0 or later is not an option, you can use -Xss to increase the JVM stack size as a workaround.)