1

When I use this dependency:

    <dependency>
        <groupId>net.manub</groupId>
        <artifactId>scalatest-embedded-kafka_2.11</artifactId>
        <version>2.0.0</version>
        <scope>test</scope>
    </dependency>

With

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql-kafka-0-10_2.11</artifactId>
        <version>2.3.0</version>
    </dependency>

I run into this error:

Cause: java.lang.ClassNotFoundException: org.apache.spark.sql.sources.v2.reader.SupportsScanUnsafeRow

Trying to figure out which version of 'scalatest-embedded-kafka' will work with Spark 2.3.

Any ideas?

DilTeam
  • 2,551
  • 9
  • 42
  • 69
  • You can check the dependencies here https://mvnrepository.com/artifact/net.manub/scalatest-embedded-kafka_2.11/2.0.0 – mazaneicha May 28 '20 at 13:03
  • I don't understand how this answer helps me solve this issue. Pleas explain which version of 'embedded kafka' I should use with 'spark-sql-kafka-0-10_2.11' version 2.3. – DilTeam May 28 '20 at 14:16
  • Sadly I think `scalatest-embedded-kafka` was dropped at version 2.0.0 and you would need a 2.3.0. Could you use a different library? – Saša Zejnilović May 28 '20 at 14:35
  • What do you mean by 'was dropped at version 2.0.0'? The 2.0.0 version works just fine with Spark 2.4 & above. I need an earlier version of 'scalatest-embedded-kafka' that works with older version of Spark (Spark 2.3). I tried 0.6.1 but that's too old. Version 1.1.1 is not available... which is confusing! – DilTeam May 28 '20 at 15:08

0 Answers0