2

How to downgrade the spark version? What could be the other solutions? I have to connect my hive tables to spark using spark session. But the spark version is not supported by zeppelin.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
SHWETA WIN
  • 41
  • 1
  • 7
  • which versions of Spark and Zeppelin are you on? Downgrading Spark would require you to just get the package and update the path. Check for other dependencies while you downgrade spark. – swamoch Aug 21 '17 at 05:29

2 Answers2

1

Here are 2 reasons.

[1] Zeppelin 0.7.2 marked spark 2.2+ as the unsupported version.

https://github.com/apache/zeppelin/blob/v0.7.2/spark/src/main/java/org/apache/zeppelin/spark/SparkVersion.java#L40

public static final SparkVersion UNSUPPORTED_FUTURE_VERSION = SPARK_2_2_0;

[2] Even if you change the const and build again, you might fail to run Zeppelin 0.7.2 with Spark 2.2

Spark dropped support for Java 7 and Zeppelin 0.7.2 was built with JDK 7. So you need to rebuild it with JDK 8.

One work around you can use is, specifying JAVA_HOME in the spark interpreter (for 2.2) as commented here

That works because only spark interpreter (for 2.2) requires java 8 and Zeppelin doesn't need it.

In short

  1. Modify here and rebuild if you want to spark 2.2 on Zeppelin branch-0.7
  2. And Use JDK 8 for the spark interpreter.
1ambda
  • 1,145
  • 8
  • 19
  • Could you explain what is the workaround? I downloaded zeppelin all 0.7.2 and I get that error "Spark 2.2.0 is not supported" with their tutorial (frustrating!). Apache Zeppelin officially supports Oracle JDK 1.7 and I have 1.8. In the picture you linked JAVA_HOME points to some directory (non existent on my machine) in jdk 1.8. – aless80 Sep 21 '17 at 15:30
  • Hi, 0.7.3 which supports spark 2.2 will be released in few days. I think you can use it. – 1ambda Sep 21 '17 at 15:47
  • 1
    thanks, in that case I'll wait for that release because this is driving me crazy. It's definitely not a good user experience when zeppelin's tutorial cannot run – aless80 Sep 21 '17 at 15:53
  • 0.7.2 has built-in support for 2.1 not 2.2 So you can run the tutorial (i checked it with 0.7.2 bin-all) https://github.com/apache/zeppelin/blob/v0.7.2/spark-dependencies/pom.xml#L535-L537 – 1ambda Sep 21 '17 at 16:39
  • The above question is about custom spark, not built-in. – 1ambda Sep 21 '17 at 16:40
0

Zeppelin 0.7.2 version supports spark 2.1.0.

Abir Chokraborty
  • 1,695
  • 4
  • 15
  • 23