1

I have successfully configured Zeppelin with Spark. However, I want to use multi-tenancy, and for that I want to configure Zeppelin with Livy and Spark.

For Livy, I provided the following two paths

export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
export HADOOP_CONF_DIR=/etc/hadoop/conf

I can run the following command successfully in Zeppelin: 

%livy.spark
sc.version

However, the following command fails: 

%livy.sql
select * from myDB.table1

I see the following error: 

<console>:14: error: not found: value sqlContext
              sqlContext.sql("select * from datalake.combination2").show(1000)

I haven't enabled Shiro authentication for Zeppelin yet. My assumption was that Livy would log into Spark using the default user as I provide the Spark home directory. Could anyone point out how can I fix the above issue? And also, what configurations would I need to do to enable impersonation?

Dan
  • 115
  • 1
  • 10

1 Answers1

0

This happens to me when my cluster don't get enough ressources to start a livy session. did you configure your livy session in zeppelin ?
interpreter -> livy2 -> basic spark configuration : 

      - livy.spark.driver.cores  
      - livy.spark.driver.memory  
      - livy.spark.executor.cores   
      - livy.spark.executor.instances   
      - livy.spark.executor.memory  
      - livy.spark.master  
      - livy.spark.submit.deployMode

also configure interpreter instantiation : globally | user | note

maxime G
  • 1,660
  • 1
  • 10
  • 27