Getting Error for org.apache.spark.sql.Encoder and missing or invalid dependency find while loading class file SQLImplicits, LowPrioritySQLImplicits

Question

I am running following code to read kafka stream with spark-3.2.2, and scala 2.12.0.

Earlier same code was working fine with spark-2.2 and scala 2.11.8,

import spark.implicits._
  val kafkaStream = spark
     .readStream
     .format("kafka")
     .option("kafka.bootstrap.servers", settings.kafka.brokers)
     .option("startingOffsets", "latest")
     .option("failOnDataLoss", "false")
     .option("subscribe", "serviceproblems")
     .load()

  val dataset = kafkaStream.select($"key", $"value").as[(String, String)]
  val mapper = new ObjectMapper
  mapper.registerModule(new ServiceProblemDeserializerModule())

Facing error while building code as below

could not find implicit value for evidence parameter of type org.apache.spark.sql.Encoder[(String, String)]
[ERROR]       val dataset = kafkaStream.select($"key", $"value").as[(String, String)]

There are some other errors as well which I am not able to understand reason as limited help is available.

[ERROR] missing or invalid dependency detected while loading class file 'SQLImplicits.class'.
Could not access type Encoder in package org.apache.spark.sql,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'SQLImplicits.class' was compiled against an incompatible version of org.apache.spark.sql.

[ERROR] missing or invalid dependency detected while loading class file 'LowPrioritySQLImplicits.class'.
Could not access type Encoder in package org.apache.spark.sql,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'LowPrioritySQLImplicits.class' was compiled against an incompatible version of org.apache.spark.sql.

[ERROR] missing or invalid dependency detected while loading class file 'package.class'.
Could not access type Row in package org.apache.spark.sql,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'package.class' was compiled against an incompatible version of org.apache.spark.sql.

[ERROR] missing or invalid dependency detected while loading class file 'Dataset.class'.
Could not access type Encoder in package org.apache.spark.sql,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Dataset.class' was compiled against an incompatible version of org.apache.spark.sql.

Appreciate any help. Thanks in Advance

Can't reproduce. The code seems to compile both in Spark 2.2.3 + Scala 2.11.8 and in Spark 3.2.2 + Scala 2.12.1 https://gist.github.com/DmytroMitin/48f0fe84ddb44a8f53ee46f7b3dc8744 https://gist.github.com/DmytroMitin/4eed7223cc1a8790342eb92f19233442 In Scala 2.12.0 I had different issues https://stackoverflow.com/questions/70931841 so try to use Scala 2.12.1+, the latest among 2.12 is 2.12.17. How do you build your project? With sbt? Try `sbt clean compile`. — Dmytro Mitin, Apr 19 '23 at 13:47
@DmytroMitin I could fix this as I found some conflicting library as mentioned below in Answer — Chandan Gawri, Apr 19 '23 at 14:00

score 0 · Accepted Answer · answered Apr 19 '23 at 13:53

After some hustle and one of Mentor support I could move away from this, what was found that we are using org.elasticsearch dependency and it also has some spark library that added conflicts, adding exclusion for duplicate dependency moved the way.

<dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch-spark-30_2.12</artifactId>
</depdendency>

Like this

<dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch-spark-30_${scala.binary.version}</artifactId>
        <version>7.12.1</version>
        <exclusions>
          <exclusion>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.12</artifactId>
          </exclusion>
          <exclusion>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.12</artifactId>
          </exclusion>
          <exclusion>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.12</artifactId>
          </exclusion>
          <exclusion>
            <groupId>org.scala.lang</groupId>
            <artifactId>scala-library_2.12.0</artifactId>
          </exclusion>
          <exclusion>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-catalyst_2.12</artifactId>
          </exclusion>
        </exclusions>
      </dependency>

Getting Error for org.apache.spark.sql.Encoder and missing or invalid dependency find while loading class file SQLImplicits, LowPrioritySQLImplicits

1 Answers1