How do I prioritise maven dependency over Spark classpath when submitting Spark job?

Question

I have a Cloudera distribution of Hadoop, Spark etc where the Spark-Kafka version is 0.8 (i.e. spark-streaming-kafka-0-8_2.11).

The issue is, version 0.8 of the Apache Spark with Kafka Integration has Kafka version 0.8.2.1 built inside and I require 0.10.0.1.

Is there a way to work around this? I do not want to use spark-streaming-kafka-0-10_2.11 because it is not a stable version.

I tried adding this to my maven dependencies (packaging with the jars) but the classpath is taking precedence over my maven dependencies.

   <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
        <version>2.3.0.cloudera1</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.kafka</groupId>
                <artifactId>kafka_2.11</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka_2.11</artifactId>
        <version>0.10.0.1</version>
    </dependency>

The old Kafka 0.8 Spark streaming client is **deprecated** in Spark 2.3+... What do you mean "not stable"? — OneCricketeer, Dec 28 '18 at 02:24

score 0 · Answer 1 · answered Dec 28 '18 at 05:40

You will need to put the kafka dependency above the spark dependency so it looks something like this:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.11</artifactId>
    <version>0.10.0.1</version>
</dependency>

<dependency>
     <groupId>org.apache.spark</groupId>
     <artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
     <version>2.3.0.cloudera1</version>
     <exclusions>
         <exclusion>
             <groupId>org.apache.kafka</groupId>
             <artifactId>kafka_2.11</artifactId>
         </exclusion>
     </exclusions>
</dependency>

How do I prioritise maven dependency over Spark classpath when submitting Spark job?

1 Answers1