0

I've been working in some scala codes to run spark streaming for kafka data acquisition. Once updated all dependencies and adding new ones to the compiled jar, I am able to to run it with spark-submit with the code spark-submit /codigos/codigos/adquisicion.jar. But when running it via livy2 I'm getting an exception when creating stream in: 

        val stream                          = KafkaUtils.createDirectStream[String, String](
        ssc,
        PreferConsistent,
        Subscribe[String, String](topics, kafkaParams)
    )

Code used to run it via livy2 is: 

uri<-"http://192.168.0.28:8999/batches"

jsonBody2 = paste0('{ "file": "/user/bigdata/adquisicion.jar",

                     "numExecutors":1,

                     "executorCores":3,

                     "name": "Adquisición Datos",

                     "args": [""],

                     "className": "codigos.adquisicionDatos" }')

request <- httr::POST(url = uri, body = jsonBody2, httr::add_headers(.headers = c('Content-Type' = 'application/json', 'X-Requested-by'='user')))

And received exception is the following one:

    18/11/06 08:01:00 ERROR ApplicationMaster: User class threw exception: java.lang.NoClassDefFoundError: org/apache/kafka/clients/consumer/Consumer
java.lang.NoClassDefFoundError: org/apache/kafka/clients/consumer/Consumer
    at org.apache.spark.streaming.kafka010.ConsumerStrategies$.Subscribe(ConsumerStrategy.scala:256)
    at codigos.adquisicionDatos$.main(adquisicionDatos.scala:251)
    at codigos.adquisicionDatos.main(adquisicionDatos.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$3.run(ApplicationMaster.scala:646)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.clients.consumer.Consumer
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 8 more

EDIT:

POM file:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>codigos</groupId>
    <artifactId>adquisicionDatos</artifactId>
    <version>1.2</version>
    <packaging>jar</packaging>

    <name>Adquisicion Datos Kafka</name>

    <properties>
        <app.main.class>adquisicionDatos.SparkKMeansApp</app.main.class>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

        <spark.core.version>2.3.2</spark.core.version>
        <!--hbase.version>0.98.7-hadoop1</hbase.version-->
        <hbase.version>1.0.1</hbase.version>
        <slf4j.version>1.7.5</slf4j.version>
        <guava.version>13.0.1</guava.version>
        <gson.version>2.2.4</gson.version>
        <spark.version>2.3.2</spark.version>
    </properties>

    <repositories>
        <repository>
            <id>scalanlp.org</id>
            <name>ScalaNLP Maven2 Repository</name>
            <url>http://repo.scalanlp.org/repo</url>
        </repository>
        <repository>
            <id>apache release</id>
            <url>https://repository.apache.org/content/repositories/releases/</url>
        </repository>
        <repository>
            <id>scala-tools.org</id>
            <name>Scala-tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </repository>
        <repository>
            <id>cloudera.repo</id>
            <url>https://repository.cloudera.com/artifactory/cloudera-repos</url>
            <name>Cloudera Repositories</name>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>scala-tools.org</id>
            <name>Scala-tools Maven2 Repository</name>
            <url>http://scala-tools.org/repo-releases</url>
        </pluginRepository>
    </pluginRepositories>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.sameersingh.scalaplot/scalaplot -->
        <dependency>
            <groupId>org.sameersingh.scalaplot</groupId>
            <artifactId>scalaplot</artifactId>
            <version>0.1</version>
        </dependency>


    <!-- https://mvnrepository.com/artifact/javax.mail/javax.mail-api -->
        <dependency>
            <groupId>javax.mail</groupId>
            <artifactId>javax.mail-api</artifactId>
            <version>1.5.1</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <!--
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        -->
        <dependency>
           <groupId>org.apache.spark</groupId>
           <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
           <version>${spark.version}</version>
           <scope>compile</scope>
           <exclusions>
               <exclusion>
                   <groupId>org.apache.kafka</groupId>
                   <artifactId>kafka_2.11</artifactId>
               </exclusion>
          </exclusions>
       </dependency>
       <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.11</artifactId>
            <version>0.10.2.1</version>
       </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql_2.11 -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>${spark.version}</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.datastax.spark/spark-cassandra-connector_2.11 -->
        <dependency>
            <groupId>com.datastax.spark</groupId>
            <artifactId>spark-cassandra-connector_2.11</artifactId>
            <version>${spark.version}</version>
            <!--<version>2.0.5</version>-->
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.postgresql/postgresql -->
        <dependency>
            <groupId>org.postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <version>42.1.1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>0.10.2.1</version>
        </dependency>





    </dependencies>

    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.scala-tools</groupId>
                    <artifactId>maven-scala-plugin</artifactId>
                    <version>2.15.2</version>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.1</version>
                    <configuration>
                        <source>1.6</source>
                        <target>1.6</target>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>

        <plugins>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                    <finalName>${project.artifactId}-${project.version}-jar-with-dependencies</finalName>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <archive>
                        <manifest>
                            <mainClass>codigos.adquisicionDatos</mainClass>
                        </manifest>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

I compile it in server and then I copy it to docker (via shared volume cp ~/adquisicionDatos/target/adquisicionDatos1.2-jar-with-dependencies.jar /mnt/codigos/adquisicion.jar) and upload it to HDFS using incrond script to automatically upload files when updated (working tested).

If running with spark-submit under root user everything is working fine but using livy or running under livy user is not working and throwing above's error.

sartions
  • 113
  • 1
  • 4
  • 15
  • You need to add the Kafka jars to Spark home on the Livy server – OneCricketeer Nov 06 '18 at 13:52
  • @cricket_007 I'm generating a 'jar-with-dependencies.jar' which includes all this jars with maven. Shouldn't that be enough? – sartions Nov 06 '18 at 14:32
  • Hmm. It should, but I've not had much experience with Livy. But it's not clear which machine this stacktrace is coming from – OneCricketeer Nov 06 '18 at 19:19
  • I don't know why but I'm completely unable to solve this. I can run it from different hosts but impossible to do it via livy. I don't know if I should upload jars in a different place but other dependencies are not in trouble. – sartions Nov 07 '18 at 12:17
  • Please refer to https://github.com/spark-jobserver/spark-jobserver/issues/911#issuecomment-355228428 – OneCricketeer Nov 07 '18 at 13:34
  • Thanks. Tried but same result. I think it has something to do with user executing jobs via livy but can't find how to deal with it – sartions Nov 07 '18 at 16:10
  • What exactly did you try? Other similar issues say the problem doesn't exist with `0_8` spark-streaming version – OneCricketeer Nov 07 '18 at 17:30
  • I tried the solution you gave in last comment and did not solve it. Actually, running by `sudo -u livy spark-submit /codigos/codigos/adquisicion.jar` returns the same error. We need 0.10+ as 0.8 has no SSL/TLS support – sartions Nov 08 '18 at 08:46
  • And you're sure that file is the same as `jar-with-dependencies.jar`? Can you show your POM? – OneCricketeer Nov 08 '18 at 15:05
  • I updated the post to show pom. It's the same, I just change name when copying file to docker and HDFS (I compile it outside docker) – sartions Nov 09 '18 at 12:23

0 Answers0