How to rebuild apache Livy with scala 2.12

Question

I'm using Spark 3.1.1 which uses Scala 2.12, and the pre-built Livy downloaded from here uses Scala 2.11 (one could find the folder named repl_2.11-jars/ after unzip).

Referred to the comment made by Aliaksandr Sasnouskikh, Livy needs to be rebuilt or it'll throw error {'msg': 'requirement failed: Cannot find Livy REPL jars.'} even in POST Session.

In the README.md, it mentioned:

By default Livy is built against Apache Spark 2.4.5

If I'd like to rebuild Livy, how could I change the spark version that it is built with?

Thanks in advance.

rmakoto · Accepted Answer · 2021-04-29T19:34:26.817

You can rebuild Livy passing spark-3.0 profile in maven to create a custom build for spark 3, for example:

git clone https://github.com/apache/incubator-livy.git && \
cd incubator-livy && \
mvn clean package -B -V -e \
        -Pspark-3.0 \
        -Pthriftserver \
        -DskipTests \
        -DskipITs \
        -Dmaven.javadoc.skip=true

This profile is defined in pom.xml, the default one installs Spark 3.0.0. You can change it to use different spark version.

<profile>
      <id>spark-3.0</id>
      <activation>
        <property>
          <name>spark-3.0</name>
        </property>
      </activation>
      <properties>
        <spark.scala-2.12.version>3.0.0</spark.scala-2.12.version>
        <spark.scala-2.11.version>2.4.5</spark.scala-2.11.version>
        <spark.version>${spark.scala-2.11.version}</spark.version>
        <netty.spark-2.12.version>4.1.47.Final</netty.spark-2.12.version>
        <netty.spark-2.11.version>4.1.47.Final</netty.spark-2.11.version>
        <netty.version>${netty.spark-2.11.version}</netty.version>
        <java.version>1.8</java.version>
        <py4j.version>0.10.9</py4j.version>
        <json4s.spark-2.11.version>3.5.3</json4s.spark-2.11.version>
        <json4s.spark-2.12.version>3.6.6</json4s.spark-2.12.version>
        <json4s.version>${json4s.spark-2.11.version}</json4s.version>
        <spark.bin.download.url>
          https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop2.7.tgz
        </spark.bin.download.url>
        <spark.bin.name>spark-3.0.0-bin-hadoop2.7</spark.bin.name>
      </properties>
    </profile>

As long as I know, Livy supports spark 3.0.x. But worth testing with 3.1.1, and let us know :)

How to change these value to build livy for spark 3.1.2, hadoop 3.2.0, scala 2.12.10? — qxk71551, Jan 12 '22 at 06:06

score 2 · Answer 2 · answered Jun 29 '21 at 06:58

2

I tried to build Livy for Spark 3.1.1 based on rmakoto's answer and it worked! I tinkered a lot and I couldn't exactly remember what I edited in the pom.xml so I am just going to attach my gist link here.

I also had to edit the python-api/pom.xml file to use Python3 to build since there's some syntax error issues when building with the default pom.xml file. Here's the pom.xml gist for python-api.

After that just build with

mvn clean package -B -V -e \
        -Pspark-3.0 \
        -Pthriftserver \
        -DskipTests \
        -DskipITs \
        -Dmaven.javadoc.skip=true

answered Jun 29 '21 at 06:58

gamberooni

51
2

Please add the codes from the gist link in the answer. – Anirban Saha Jun 29 '21 at 08:31
1

I tried to add it into the answer but its over 30000 characters so I couldn't :) – gamberooni Jun 29 '21 at 08:46
I tried your guide, but i dont know where output stored after built. How to package it same zip package on home page livy? – qxk71551 Jan 12 '22 at 06:02
Have needed to change these config to scala 2.12 2.11 ${scala-2.11.version} ${spark.scala-2.11.version} – qxk71551 Jan 12 '22 at 06:05
I try to build for spark 3.1.2, hadoop 3.2.0, scala 2.12.10 – qxk71551 Jan 12 '22 at 06:05
I am sorry I did this at my previous company and (almost) all was forgotten by now. What I still vaguely remember is that after you run the maven build command, there should be additional jars created in the directories (e.g. client-common, client-http, thriftserver and etc.). I think what I did was to download the Apache Livy package from the official website, use that as a reference to see what jars need to be present. – gamberooni Jan 20 '22 at 12:08
After that I just made it look as similar to the one I downloaded from the official website, but with the new jars. – gamberooni Jan 20 '22 at 12:09

Herrmann · Answer 3 · 2021-06-30T03:25:43.020

Based on @gamberooni 's changes (but using 3.1.2 instead of 3.1.1 for the Spark version and Hadoop 3.2.0 instead of 3.2.1), this is the diff:

diff --git a/pom.xml b/pom.xml
index d2e535a..5c28ee6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -79,12 +79,12 @@
 
   <properties>
     <asynchttpclient.version>2.10.1</asynchttpclient.version>
-    <hadoop.version>2.7.3</hadoop.version>
+    <hadoop.version>3.2.0</hadoop.version>
     <hadoop.scope>compile</hadoop.scope>
     <spark.scala-2.11.version>2.4.5</spark.scala-2.11.version>
-    <spark.scala-2.12.version>2.4.5</spark.scala-2.12.version>
-    <spark.version>${spark.scala-2.11.version}</spark.version>
-    <hive.version>3.0.0</hive.version>
+    <spark.scala-2.12.version>3.1.2</spark.scala-2.12.version>
+    <spark.version>${spark.scala-2.12.version}</spark.version>
+    <hive.version>3.1.2</hive.version>
     <commons-codec.version>1.9</commons-codec.version>
     <httpclient.version>4.5.3</httpclient.version>
     <httpcore.version>4.4.4</httpcore.version>
@@ -1060,7 +1060,7 @@
         </property>
       </activation>
       <properties>
-        <spark.scala-2.12.version>3.0.0</spark.scala-2.12.version>
+        <spark.scala-2.12.version>3.1.2</spark.scala-2.12.version>
         <spark.scala-2.11.version>2.4.5</spark.scala-2.11.version>
         <spark.version>${spark.scala-2.11.version}</spark.version>
         <netty.spark-2.12.version>4.1.47.Final</netty.spark-2.12.version>
@@ -1072,9 +1072,9 @@
         <json4s.spark-2.12.version>3.6.6</json4s.spark-2.12.version>
         <json4s.version>${json4s.spark-2.11.version}</json4s.version>
         <spark.bin.download.url>
-          https://archive.apache.org/dist/spark/spark-3.0.0/spark-3.0.0-bin-hadoop2.7.tgz
+          https://archive.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
         </spark.bin.download.url>
-        <spark.bin.name>spark-3.0.0-bin-hadoop2.7</spark.bin.name>
+        <spark.bin.name>spark-3.1.2-bin-hadoop3.2</spark.bin.name>
       </properties>
     </profile>
 
diff --git a/python-api/pom.xml b/python-api/pom.xml
index 8e5cdab..a8fb042 100644
--- a/python-api/pom.xml
+++ b/python-api/pom.xml
@@ -46,7 +46,7 @@
               <goal>exec</goal>
             </goals>
             <configuration>
-            <executable>python</executable>
+            <executable>python3</executable>
               <arguments>
                 <argument>setup.py</argument>
                 <argument>sdist</argument>
@@ -60,7 +60,7 @@
               <goal>exec</goal>
             </goals>
             <configuration>
-              <executable>python</executable>
+              <executable>python3</executable>
               <skip>${skipTests}</skip>
               <arguments>
                 <argument>setup.py</argument>

Thanks. I built the package successfully (using mvn package). How do I create a package to add to say /opt/livy ? Or can we just run Livy directly from the package directory? — tallharish, Apr 14 '22 at 23:59

study · Answer 4 · 2023-01-05T03:40:49.323

0

My Spark version is 3.2.1, and my scala version is 2.12.15. I have successfully built and put it into use. I will show my construction process. Pull the master code of livy and modify the pom file as follows：