Null Pointer Exception After Adding Dependency to Spark2 Interpreter

Question

I am trying to add this dependency to the spark 2 interpreter in zeppelin

https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.11/2.2.0

However, after adding the dependency, I get a null pointer exception when running any code.

Null Pointer Exception Screenshot

Spark & Scala Version Screenshot

Adding Dependency

[Please do not post images of text in a question](https://meta.stackoverflow.com/questions/303812/discourage-screenshots-of-code-and-or-errors), as these are not searchable. Paste the text itself. Then people can find it to give you an answer. — Brian Tompsett - 汤莱恩, Oct 06 '18 at 17:00

score 0 · Answer 1 · answered Oct 07 '18 at 13:20

0

You don't need to add spark-sql, it is already in spark interpreter.

answered Oct 07 '18 at 13:20

zjffdu

25,496
45
109
159

Im trying to run this command: data=fire_services_customDF.withColumn("CallDateTmp",date_format(to_date(col("CallDate"), "MM/dd/yy"), "yyyy-MM-dd")).cast("timestamp")Getting this error: :37: error: not found: value date_format data=fire_services_customDF.withColumn("CallDateTmp",date_format(to_date(col("CallDate"), "MM/dd/yy"), "yyyy-MM-dd")).cast("timestamp") :37: error: not found: value to_date data=fire_services_customDF.withColumn("CallDateTmp",date_format(to_date(col("CallDate"), "MM/dd/yy"), "yyyy-MM-dd")).cast("timestamp") – Mustafa Akmal Oct 08 '18 at 06:58
date_format is the function of spark sql. You need to import them explicitly. This error is not due to the missing of spark sql jar – zjffdu Oct 08 '18 at 11:54

score 0 · Answer 2 · answered Oct 07 '18 at 18:19

0

Just add %spark.sql at the top of your notebook to provide an SQL environment

https://zeppelin.apache.org/docs/0.8.0/interpreter/spark.html#overview

answered Oct 07 '18 at 18:19

bp2010

2,342
17
34

Getting this error now: :2: error: ';' expected but '.' found. %spark.sql – Mustafa Akmal Oct 08 '18 at 06:54
make sure the correct interpretors are selected (the wheel on the right hand side) https://stackoverflow.com/a/33567972/3213111 – bp2010 Oct 08 '18 at 09:05

score 0 · Accepted Answer · answered Oct 08 '18 at 07:20

I solved the problem. I was defining a class in Scala. The methods to_date & date_format were being used inside the class but my import statements were outside the class. All I had to do was place the import statements inside the class brackets and it worked fine.

case class HelperClass(){
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._

var fire_services_customDF = fire_servicesDF
var data=fire_servicesDF

def SetDatatypes() : Unit = {
    data=fire_services_customDF.withColumn("CallDateTmp",date_format(to_date(col("CallDate"), "MM/dd/yy"), "yyyy-MM-dd").cast("timestamp"))
}

def PrintSchema() : Unit= {
         data.printSchema
}

}

Null Pointer Exception After Adding Dependency to Spark2 Interpreter

3 Answers3